cardano-testnet | Fail startNode early on exit when waiting for socket

[ch1bo]

Includes #6405

This dramatically reduces the feedback loop in case the cardano-node exits early.

Note that this only checks for node shutdown initially and not changes how processes are observed after the testnet is deemed to be running.

For example, before using a failing cardano-node:

λ time (cardano-testnet create-env --output devnet --num-pool-nodes 3 --slot-length 1 --testnet-magic 164 --params-mainnet
cat devnet/configuration.yaml | jq '.EnableP2P = true' > devnet/configuration-p2p.yaml && mv devnet/configuration{-p2p,}.yaml
cardano-testnet cardano --node-env devnet --update-time)

Starting testnet in environment: /home/ch1bo/code/iog/ouroboros-leios/demo/proto-devnet/devnet/
cardano-testnet: UnliftIO.Exception.throwString called with:

Some nodes failed to start:
Cardano node process did not start: You must define the LEIOS_DB_PATH variable for this demo.
cardano-node: ExceptionInLinkedThread "ThreadId 38" (ExitFailure 1)

Cardano node process did not start: You must define the LEIOS_DB_PATH variable for this demo.
cardano-node: ExceptionInLinkedThread "ThreadId 36" (ExitFailure 1)

Cardano node process did not start: You must define the LEIOS_DB_PATH variable for this demo.
cardano-node: ExceptionInLinkedThread "ThreadId 36" (ExitFailure 1)

Called from:
  throwString (src/Testnet/Start/Cardano.hs:347:5 in cardano-testnet-10.0.1-8tGdaJ4pSa97bIgUjr90jP:Testnet.Start.Cardano)
  cardanoTestnet (src/Parsers/Run.hs:98:18 in cardano-testnet-10.0.1-8tGdaJ4pSa97bIgUjr90jP:Parsers.Run)

0.50s user 0.31s system 0% cpu 2:00.54 total

And after:

λ time (cardano-testnet create-env --output devnet --num-pool-nodes 3 --slot-length 1 --testnet-magic 164 --params-mainnet
cat devnet/configuration.yaml | jq '.EnableP2P = true' > devnet/configuration-p2p.yaml && mv devnet/configuration{-p2p,}.yaml
cardano-testnet cardano --node-env devnet --update-time)

Starting testnet in environment: /home/ch1bo/code/iog/ouroboros-leios/demo/proto-devnet/devnet/
cardano-testnet: UnliftIO.Exception.throwString called with:

Some nodes failed to start:
Cardano node process did not start: You must define the LEIOS_DB_PATH variable for this demo.
cardano-node: ExceptionInLinkedThread "ThreadId 38" (ExitFailure 1)

Cardano node process did not start: You must define the LEIOS_DB_PATH variable for this demo.
cardano-node: ExceptionInLinkedThread "ThreadId 36" (ExitFailure 1)

Cardano node process did not start: You must define the LEIOS_DB_PATH variable for this demo.
cardano-node: ExceptionInLinkedThread "ThreadId 36" (ExitFailure 1)

Called from:
  throwString (src/Testnet/Start/Cardano.hs:347:5 in cardano-testnet-10.0.1-inplace:Testnet.Start.Cardano)
  cardanoTestnet (src/Parsers/Run.hs:98:18 in cardano-testnet-10.0.1-inplace:Parsers.Run)

0.39s user 0.19s system 3% cpu 15.452 total

Checklist

• Commit sequence broadly makes sense and commits have useful messages
• New tests are added if needed and existing tests are updated.
• Any changes are noted in the CHANGELOG.md for affected package
• The version bounds in .cabal files are updated
• Self-reviewed the diff

[ch1bo]

@carbolymer @Jimbo4350 I was considering observing the process for exit in a background thread the whole MonadResource life time over (using the already existing asyncRegister_), but was not sure what the general behavior of cardano-testnet should be eventually? Note that this would only be able to bring down the main thread via link and async exceptions. AFAIK the only alternative for nicely dealing with this would be using the with..-pattern everywhere (but that is a major refactor away from runResource?)

[carbolymer]

@ch1bo Why not use a similar approach in your waitForShutdown in #6405 - I mean just use waitForProcess there too?

[ch1bo]

@ch1bo Why not use a similar approach in your waitForShutdown in #6405 - I mean just use waitForProcess there too?

waitForShutdown is just an idle loop to keep the main thread alive. The unwrapping of ResourceT will take care of cleaning up (from what I saw so far, maybe there is some signal forwarding too involved).