python asyncio - how to wait for a cancelled shielded task?

Question

If I have a coroutine which runs a task which should not be cancelled, I will wrap that task in asyncio.shield().

It seems the behavior of cancel

Answer 1

It seems the behavior of cancel and shield is not what I would expect. If I have a task wrapped in shield and I cancel it, the await-ing coroutine returns from that await statement immediately rather than awaiting for the task to finish as shield would suggest. Additionally, the task that was run with shield continues to run but its future is now cancelled an not await-able.

Conceptually shield is like a bullet-proof vest that absorbs the bullet and protects the wearer, but is itself destroyed by the impact. shield absorbs the cancellation, and reports itself as canceled, raising a CancelledError when asked for result, but allows the protected task to continue running. (Artemiy's answer explains the implementation.)

Cancellation of the future returned by shield could have been implemented differently, e.g. by completely ignoring the cancel request. The current approach ensures that the cancellation "succeeds", i.e. that the canceller can't tell that the cancellation was in fact circumvented. This is by design, and it makes the cancellation mechanism more consistent on the whole.

What is the proper method to shield a task from cancellation and then wait for it to complete before returning

By keeping two objects: the original task, and the shielded task. You pass the shielded task to whatever function it is that might end up canceling it, and you await the original one. For example:

async def coro():
    print('starting')
    await asyncio.sleep(2)
    print('done sleep')

async def cancel_it(some_task):
    await asyncio.sleep(0.5)
    some_task.cancel()
    print('cancellation effected')

async def main():
    loop = asyncio.get_event_loop()
    real_task = loop.create_task(coro())
    shield = asyncio.shield(real_task)
    # cancel the shield in the background while we're waiting
    loop.create_task(cancel_it(shield))
    await real_task

    assert not real_task.cancelled()
    assert shield.cancelled()

asyncio.get_event_loop().run_until_complete(main())

The code waits for the task to fully complete, despite its shield getting cancelled.

Answer 2

It would make more sense if asyncio.shield() raised the asyncio.CancelledError after the await-ed task has completed, but obviously there is some other idea going on here that I don't understand.

asyncio.shield

• creates a dummy future, that may be cancelled
• executes the wrapped coroutine as future and bind to it a callback on done to setting a result for the dummy future from the completed wrapped coroutine
• returns the dummy future

You can see the implementation here

What is the proper method to shield a task from cancellation and then wait for it to complete before returning

You should shield count(5) future

async def t():
  c_ft = asyncio.ensure_future(count(5))
  try:
    await asyncio.shield(c_ft)
  except asyncio.CancelledError:
    print('This gets called at 3, not 5')
    await c_ft

  return 42

or t() future

async def t():
  await count(5)
  return 42    

async def m():
  ft = asyncio.ensure_future(t())
  shielded_ft = asyncio.shield(ft)
  ct = asyncio.ensure_future(c(shielded_ft))

  try:
    r = await shielded_ft
  except asyncio.CancelledError:
    print('Shield cancelled')
    r = await ft