Creating dynamically named variables in a function in python 3 / Understanding exec / eval / locals in python 3

Question

First of all, let me say that I read the many threads with similar topics on creating dynamically named variables, but they mostly relate to Python 2 or they assume you are

Answer 1

On the exec/eval/locals question

At least on the CPython implementation modifications to the locals() dictionary do not actually change the names in the local scope, which is why it's meant to be used read-only. You can change it, and you can see your changes in the dictionary object, but the actual local scope is not changed.

exec() takes two optional dictionary arguments, a global scope and a local scope. It defaults to globals() and locals(), but since changes to locals() aren't "real" outside of the dictionary, exec() only affects the "real" local scope when globals() is locals(), i.e. in a module outside of any function. (So in your case it's failing because it's inside a function scope).

The "better" way to use exec() in this case is to pass in your own dictionary, then operate on the values in that.

def foo():
    exec_scope = {}
    exec("y = 2", exec_scope)
    print(exec_scope['y'])
foo()

In this case, exec_scope is used as the global and local scope for the exec, and after the exec it will contain {'y': 2, '__builtins__': __builtins__} (the builtins are inserted for you if not present)

If you want access to more globals you could do exec_scope = dict(globals()).

Passing in different global and local scope dictionaries can produce "interesting" behavior.

If you pass the same dictionary into successive calls to exec or eval, then they have the same scope, which is why your eval worked (it implicitly used the locals() dictionary).

On dynamic variable names

If you set the name from a string, what's so wrong about getting the value as a string (i.e. what a dictionary does)? In other words, why would you want to set locals()['K'] and then access K? If K is in your source it's not really a dynamically set name... hence, dictionaries.

Answer 2

When you're not sure why something works the way it does in Python, it often can help to put the behavior that you're confused by in a function and then disassemble it from the Python bytecode with the dis module.

Lets start with a simpler version of your code:

def foo():
    exec("K = 89")
    print(K)

If you run foo(), you'll get the same exception you're seeing with your more complicated function:

>>> foo()
Traceback (most recent call last):
  File "<pyshell#167>", line 1, in <module>
    foo()
  File "<pyshell#166>", line 3, in foo
    print(K)
NameError: name 'K' is not defined

Lets disassemble it and see why:

>>> import dis
>>> dis.dis(foo)
  2           0 LOAD_GLOBAL              0 (exec)
              3 LOAD_CONST               1 ('K = 89')
              6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
              9 POP_TOP

  3          10 LOAD_GLOBAL              1 (print)
             13 LOAD_GLOBAL              2 (K)
             16 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             19 POP_TOP
             20 LOAD_CONST               0 (None)
             23 RETURN_VALUE

The operation that you need to pay attention to is the one labeled "13". This is where the compiler handles looking up K within the last line of the function (print(K)). It is using the LOAD_GLOBAL opcode, which fails because "K" is not a global variable name, rather it's a value in our locals() dict (added by the exec call).

What if we persuaded the compiler to see K as a local variable (by giving it a value before running the exec), so it will know not to look for a global variable that doesn't exist?

def bar():
    K = None
    exec("K = 89")
    print(K)

This function won't give you an error if you run it, but you won't get the expected value printed out:

>>> bar()
None

Lets disassemble to see why:

>>> dis.dis(bar)
  2           0 LOAD_CONST               0 (None)
              3 STORE_FAST               0 (K)

  3           6 LOAD_GLOBAL              0 (exec)
              9 LOAD_CONST               1 ('K = 89')
             12 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             15 POP_TOP

  4          16 LOAD_GLOBAL              1 (print)
             19 LOAD_FAST                0 (K)
             22 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             25 POP_TOP
             26 LOAD_CONST               0 (None)
             29 RETURN_VALUE

Note the opcodes used at "3" and "19". The Python compiler uses STORE_FAST and LOAD_FAST to put the value for the local variable K into slot 0 and later fetch it back out. Using numbered slots is significantly faster than inserting and fetching values from a dictionary like locals(), which is why the Python compiler does it for all local variable access in a function. You can't overwrite a local variable in a slot by modifying the dictionary returned by locals() (as exec does, if you don't pass it a dict to use for its namespace).

Indeed, lets try a third version of our function, where we peek into locals again when we have K defined as a regular local variable:

def baz():
    K = None
    exec("K = 89")
    print(locals())

You won't see 89 in the output this time either!

>>> baz()
{"K": None}

The reason you see the old K value in locals() is explained in the function's documentation:

Update and return a dictionary representing the current local symbol table.

The slot that the local variable K's value is stored in was not changed by the exec statement, which only modifies the locals() dict. When you call locals() again, Python "update[s]" the dictionary with the value from the slot, replacing the value stored there by exec.

This is why the docs go on to say:

Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter.

Your exec call is modifying the locals() dict, and you're finding how its changes are not always seen by your later code.