About the changing id of an immutable string

Question

Something about the id of objects of type str (in python 2.7) puzzles me. The str type is immutable, so I would expect that once it is

Answer 1

This behavior is specific to the Python interactive shell. If I put the following in a .py file:

print id('so')
print id('so')
print id('so')

and execute it, I receive the following output:

2888960
2888960
2888960

In CPython, a string literal is treated as a constant, which we can see in the bytecode of the snippet above:

  2           0 LOAD_GLOBAL              0 (id)
              3 LOAD_CONST               1 ('so')
              6 CALL_FUNCTION            1
              9 PRINT_ITEM          
             10 PRINT_NEWLINE       

  3          11 LOAD_GLOBAL              0 (id)
             14 LOAD_CONST               1 ('so')
             17 CALL_FUNCTION            1
             20 PRINT_ITEM          
             21 PRINT_NEWLINE       

  4          22 LOAD_GLOBAL              0 (id)
             25 LOAD_CONST               1 ('so')
             28 CALL_FUNCTION            1
             31 PRINT_ITEM          
             32 PRINT_NEWLINE       
             33 LOAD_CONST               0 (None)
             36 RETURN_VALUE  

The same constant (i.e. the same string object) is loaded 3 times, so the IDs are the same.

Answer 2

So while Python is not guaranteed to intern strings, it will frequently reuse the same string, and is may mislead. It's important to know that you shouldn't check id or is for equality of strings.

To demonstrate this, one way I've discovered to force a new string in Python 2.6 at least:

>>> so = 'so'
>>> new_so = '{0}'.format(so)
>>> so is new_so 
False

and here's a bit more Python exploration:

>>> id(so)
102596064
>>> id(new_so)
259679968
>>> so == new_so
True

Answer 3

In your first example a new instance of the string 'so' is created each time, hence different id.

In the second example you are binding the string to a variable and Python can then maintain a shared copy of the string.

Answer 4

A more simplified way to understand the behaviour is to check the following Data Types and Variables.

Section "A String Pecularity" illustrates your question using special characters as example.

Answer 5

CPython does not promise to intern all strings by default, but in practice, a lot of places in the Python codebase do reuse already-created string objects. A lot of Python internals use (the C-equivalent of) the sys.intern() function call to explicitly intern Python strings, but unless you hit one of those special cases, two identical Python string literals will produce different strings.

Python is also free to reuse memory locations, and Python will also optimize immutable literals by storing them once, at compile time, with the bytecode in code objects. The Python REPL (interactive interpreter) also stores the most recent expression result in the _ name, which muddles up things some more.

As such, you will see the same id crop up from time to time.

Running just the line id(<string literal>) in the REPL goes through several steps:

1. The line is compiled, which includes creating a constant for the string object:

   >>> compile("id('foo')", '<stdin>', 'single').co_consts
   ('foo', None)
   

   This shows the stored constants with the compiled bytecode; in this case a string 'foo' and the None singleton. Simple expressions consisting of that produce an immutable value may be optimised at this stage, see the note on optimizers, below.

2. On execution, the string is loaded from the code constants, and id() returns the memory location. The resulting int value is bound to _, as well as printed:

   >>> import dis
   >>> dis.dis(compile("id('foo')", '<stdin>', 'single'))
     1           0 LOAD_NAME                0 (id)
                 3 LOAD_CONST               0 ('foo')
                 6 CALL_FUNCTION            1
                 9 PRINT_EXPR          
                10 LOAD_CONST               1 (None)
                13 RETURN_VALUE        
   

3. The code object is not referenced by anything, reference count drops to 0 and the code object is deleted. As a consequence, so is the string object.

Python can then perhaps reuse the same memory location for a new string object, if you re-run the same code. This usually leads to the same memory address being printed if you repeat this code. This does depend on what else you do with your Python memory.

ID reuse is not predictable; if in the meantime the garbage collector runs to clear circular references, other memory could be freed and you'll get new memory addresses.

Next, the Python compiler will also intern any Python string stored as a constant, provided it looks enough like a valid identifier. The Python code object factory function PyCode_New will intern any string object that contains only ASCII letters, digits or underscores, by calling intern_string_constants(). This function recurses through the constants structures and for any string object v found there executes:

if (all_name_chars(v)) {
    PyObject *w = v;
    PyUnicode_InternInPlace(&v);
    if (w != v) {
        PyTuple_SET_ITEM(tuple, i, v);
        modified = 1;
    }
}

where all_name_chars() is documented as

/* all_name_chars(s): true iff s matches [a-zA-Z0-9_]* */

Since you created strings that fit that criterion, they are interned, which is why you see the same ID being used for the 'so' string in your second test: as long as a reference to the interned version survives, interning will cause future 'so' literals to reuse the interned string object, even in new code blocks and bound to different identifiers. In your first test, you don't save a reference to the string, so the interned strings are discarded before they can be reused.

Incidentally, your new name so = 'so' binds a string to a name that contains the same characters. In other words, you are creating a global whose name and value are equal. As Python interns both identifiers and qualifying constants, you end up using the same string object for both the identifier and its value:

>>> compile("so = 'so'", '<stdin>', 'single').co_names[0] is compile("so = 'so'", '<stdin>', 'single').co_consts[0]
True

If you create strings that are either not code object constants, or contain characters outside of the letters + numbers + underscore range, you'll see the id() value not being reused:

>>> some_var = 'Look ma, spaces and punctuation!'
>>> some_other_var = 'Look ma, spaces and punctuation!'
>>> id(some_var)
4493058384
>>> id(some_other_var)
4493058456
>>> foo = 'Concatenating_' + 'also_helps_if_long_enough'
>>> bar = 'Concatenating_' + 'also_helps_if_long_enough'
>>> foo is bar
False
>>> foo == bar
True

The Python compiler either uses the peephole optimizer (Python versions < 3.7) or the more capable AST optimizer (3.7 and newer) to pre-calculate (fold) the results of simple expressions involving constants. The peepholder limits it's output to a sequence of length 20 or less (to prevent bloating code objects and memory use), while the AST optimizer uses a separate limit for strings of 4096 characters. This means that concatenating shorter strings consisting only of name characters can still lead to interned strings if the resulting string fits within the optimizer limits of your current Python version.

E.g. on Python 3.7, 'foo' * 20 will result in a single interned string, because constant folding turns this into a single value, while on Python 3.6 or older only 'foo' * 6 would be folded:

>>> import dis, sys
>>> sys.version_info
sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
>>> dis.dis("'foo' * 20")
  1           0 LOAD_CONST               0 ('foofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoo')
              2 RETURN_VALUE

and

>>> dis.dis("'foo' * 6")
  1           0 LOAD_CONST               2 ('foofoofoofoofoofoo')
              2 RETURN_VALUE
>>> dis.dis("'foo' * 7")
  1           0 LOAD_CONST               0 ('foo')
              2 LOAD_CONST               1 (7)
              4 BINARY_MULTIPLY
              6 RETURN_VALUE