My problem is that when a Task has a Task.WhenAll() call (running other Tasks) the line of WhenAll() makes the consuming code continue execution, unlike what I would expect.
Task.WhenAll returns a new Task immediately, it does not block. The returned task will complete when all tasks passed to WhenAll
have completed.
It is an asynchronous equivalent to Task.WaitAll, and this is the method to use if you want to block.
However you have another problem. Using Task.Factory.StartNew and passing an async
delegate seems to lead to a type of Task<Task>
where the outer task completes when the inner task starts to execute (rather than when it has completed).
Using the newer Task.Run avoids this.