Why does python asyncio loop.call_soon overwrite data?

Question

I created a hard to track down bug in our code, but do not understand why it occurs. The problem occurs when pushing the same async function multiple times to call soon. It d

Answer 1

In addition to correct explanations by others concerning the error in the lambda, also note that you don't even need the lambda. Since do_something is a coroutine, just calling it will not execute any of its code until the next iteration of the event loop, so you automatically have the effect of a call_soon. (This is analogous to how calling a generator function doesn't start executing it until you start exhausing the returned iterator.)

In other words, you can replace

self.loop.call_soon(lambda: asyncio.ensure_future(self.do_something(k, v)))

with the simpler

self.loop.create_task(self.do_something(k, v))

create_task is preferable to ensure_future when you are dealing with a coroutine.

Answer 2

This has nothing to do with the async code, but with the lambda you're creating in your loop. When you write lambda: asyncio.ensure_future(self.do_something(k, v)), you're creating a closure that accesses the variables k and v from the enclosing namespace (and self too, but that's not a problem). When the lambda function is called, it will use the values bound by those names in the outer scope at that time of the call, not the values they had when the lambda was defined. Since k and v change value on each iteration of the loop, that's causing all the lambda functions to see the same values (the last ones).

A common way to avoid this issue is to make the current values of the variables default values for arguments to the lambda function:

self.loop.call_soon(lambda k=k, v=v: asyncio.ensure_future(self.do_something(k, v)))

Answer 3

Your problem actually has nothing to do with asyncio. The k and v in lambda: asyncio.ensure_future(self.do_something(k, v)) still refer to the variables in your outer scope. Their values change by the time you call your function:

i = 1
f = lambda: print(i)

f()  # 1
i = 2
f()  # 2

A common solution is to define your function and (ab)use default arguments to create a variable local to your function that holds the value of i at the time the function was created, not called:

i = 1
f = lambda i=i: print(i)

f()  # 1
i = 2
f()  # 1

You can use f = lambda x=i: print(x) if the naming confuses you.