Python name mangling

Question

In other languages, a general guideline that helps produce better code is always make everything as hidden as possible. If in doubt about whether a variable should be privat

Answer 1

I wouldn't say that practice produces better code. Visibility modifiers only distract you from the task at hand, and as a side effect force your interface to be used as you intended. Generally speaking, enforcing visibility prevents programmers from messing things up if they haven't read the documentation properly.

A far better solution is the route that Python encourages: Your classes and variables should be well documented, and their behaviour clear. The source should be available. This is far more extensible and reliable way to write code.

My strategy in Python is this:

1. Just write the damn thing, make no assumptions about how your data should be protected. This assumes that you write to create the ideal interfaces for your problems.
2. Use a leading underscore for stuff that probably won't be used externally, and isn't part of the normal "client code" interface.
3. Use double underscore only for things that are purely convenience inside the class, or will cause considerable damage if accidentally exposed.

Above all, it should be clear what everything does. Document it if someone else will be using it. Document it if you want it to be useful in a year's time.

As a side note, you should actually be going with protected in those other languages: You never know your class might be inherited later and for what it might be used. Best to only protect those variables that you are certain cannot or should not be used by foreign code.

Answer 2

When in doubt, leave it "public" - I mean, do not add anything to obscure the name of your attribute. If you have a class with some internal value, do not bother about it. Instead of writing:

class Stack(object):

    def __init__(self):
        self.__storage = [] # Too uptight

    def push(self, value):
        self.__storage.append(value)

write this by default:

class Stack(object):

    def __init__(self):
        self.storage = [] # No mangling

    def push(self, value):
        self.storage.append(value)

This is for sure a controversial way of doing things. Python newbies just hate it and even some old Python guys despise this default - but it is the default anyway, so I really recommend you to follow it, even if you feel uncomfortable.

If you really want to send the message "Can't touch this!" to your users, the usual way is to precede the variable with one underscore. This is just a convention, but people understand it and take double care when dealing with such stuff:

class Stack(object):

    def __init__(self):
        self._storage = [] # This is ok but pythonistas use it to be relaxed about it

    def push(self, value):
        self._storage.append(value)

This can be useful, too, for avoiding conflict between property names and attribute names:

 class Person(object):
     def __init__(self, name, age):
         self.name = name
         self._age = age if age >= 0 else 0

     @property
     def age(self):
         return self._age

     @age.setter
     def age(self, age):
         if age >= 0:
             self._age = age
         else:
             self._age  = 0

What about the double underscore? Well, the double underscore magic is used mainly to avoid accidental overloading of methods and name conflicts with superclasses' attributes. It can be quite useful if you write a class that is expected to be extended many times.

If you want to use it for other purposes, you can, but it is neither usual nor recommended.

EDIT: Why is this so? Well, the usual Python style does not emphasize making things private - on the contrary! There are a lot of reasons for that - most of them controversial... Let us see some of them.

Python has properties

Most OO languages today use the opposite approach: what should not be used should not be visible, so attributes should be private. Theoretically, this would yield more manageable, less coupled classes, because no one would change values inside the objects recklessly.

However, it is not so simple. For example, Java classes do have a lot attributes and getters that just get the values and setters that just set the values. You need, let us say, seven lines of code to declare a single attribute - which a Python programmer would say is needlessly complex. Also, in practice, you just write this whole lot of code to get one public field, since you can change its value using the getters and setters.

So why to follow this private-by-default policy? Just make your attributes public by default. Of course, this is problematic in Java, because if you decide to add some validation to your attribute, it would require you to change all

person.age = age;

in your code to, let us say,

person.setAge(age);

setAge() being:

public void setAge(int age) {
    if (age >= 0) {
        this.age = age;
    } else {
        this.age = 0;
    }
}

So in Java (and other languages), the default is to use getters and setters anyway, because they can be annoying to write but can spare you a lot of time if you find yourself in the situation I've described.

However, you do not need to do it in Python, since Python has properties. If you have this class:

 class Person(object):
     def __init__(self, name, age):
         self.name = name
         self.age = age

and then you decide to validate ages, you do not need to change the person.age = age pieces of your code. Just add a property (as shown below)

 class Person(object):
     def __init__(self, name, age):
         self.name = name
         self._age = age if age >= 0 else 0

     @property
     def age(self):
         return self._age

     @age.setter
     def age(self, age):
         if age >= 0:
             self._age = age
         else:
             self._age  = 0

If you can do it and still use person.age = age, why would you add private fields and getters and setters?

(Also, see Python is not Java and this article about the harms of using getters and setters.).

Everything is visible anyway - and trying to hide just complicates your work

Even in languages where there are private attributes, you can access them through some kind of reflection/introspection library. And people do it a lot, in frameworks and for solving urgent needs. The problem is that introspection libraries are just a hard way of doing what you could do with public attributes.

Since Python is a very dynamic language, it is just counterproductive to add this burden to your classes.

The problem is not being possible to see - it is being required to see

For a Pythonista, encapsulation is not the inability of seeing the internals of classes, but the possibility of avoiding looking at it. What I mean is, encapsulation is the property of a component which allows it to be used without the user being concerned about the internal details. If you can use a component without bothering yourself about its implementation, then it is encapsulated (in the opinion of a Python programmer).

Now, if you wrote your class in such a way you can use it without having to think about implementation details, there is no problem if you want to look inside the class for some reason. The point is: your API should be good and the rest is details.

Guido said so

Well, this is not controversial: he said so, actually. (Look for "open kimono.")

This is culture

Yes, there are some reasons, but no critical reason. This is mostly a cultural aspect of programming in Python. Frankly, it could be the other way, too - but it is not. Also, you could just as easily ask the other way around: why do some languages use private attributes by default? For the same main reason as for the Python practice: because it is the culture of these languages, and each choice has advantages and disadvantages.

Since there already is this culture, you are well advised to follow it. Otherwise, you will get annoyed by Python programmers telling you to remove the __ from your code when you ask a question in Stack Overflow :)

Answer 3

First - What is name mangling?

Name mangling is invoked when you are in a class definition and use __any_name or __any_name_, that is, two (or more) leading underscores and at most one trailing underscore.

class Demo:
    __any_name = "__any_name"
    __any_other_name_ = "__any_other_name_"

And now:

>>> [n for n in dir(Demo) if 'any' in n]
['_Demo__any_name', '_Demo__any_other_name_']
>>> Demo._Demo__any_name
'__any_name'
>>> Demo._Demo__any_other_name_
'__any_other_name_'

When in doubt, do what?

The ostensible use is to prevent subclassers from using an attribute that the class uses.

A potential value is in avoiding name collisions with subclassers who want to override behavior, so that the parent class functionality keeps working as expected. However, the example in the Python documentation is not Liskov substitutable, and no examples come to mind where I have found this useful.

The downsides are that it increases cognitive load for reading and understanding a code base, and especially so when debugging where you see the double underscore name in the source and a mangled name in the debugger.

My personal approach is to intentionally avoid it. I work on a very large code base. The rare uses of it stick out like a sore thumb and do not seem justified.

You do need to be aware of it so you know it when you see it.

PEP 8

PEP 8, the Python standard library style guide, currently says (abridged):

There is some controversy about the use of __names.

If your class is intended to be subclassed, and you have attributes that you do not want subclasses to use, consider naming them with double leading underscores and no trailing underscores.

1. Note that only the simple class name is used in the mangled name, so if a subclass chooses both the same class name and attribute name, you can still get name collisions.

2. Name mangling can make certain uses, such as debugging and __getattr__() , less convenient. However the name mangling algorithm is well documented and easy to perform manually.

3. Not everyone likes name mangling. Try to balance the need to avoid accidental name clashes with potential use by advanced callers.

How does it work?

If you prepend two underscores (without ending double-underscores) in a class definition, the name will be mangled, and an underscore followed by the class name will be prepended on the object:

>>> class Foo(object):
...     __foobar = None
...     _foobaz = None
...     __fooquux__ = None
... 
>>> [name for name in dir(Foo) if 'foo' in name]
['_Foo__foobar', '__fooquux__', '_foobaz']

Note that names will only get mangled when the class definition is parsed:

>>> Foo.__test = None
>>> Foo.__test
>>> Foo._Foo__test
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'Foo' has no attribute '_Foo__test'

Also, those new to Python sometimes have trouble understanding what's going on when they can't manually access a name they see defined in a class definition. This is not a strong reason against it, but it's something to consider if you have a learning audience.

One Underscore?

If the convention is to use only one underscore, I'd also like to know the rationale.

When my intention is for users to keep their hands off an attribute, I tend to only use the one underscore, but that's because in my mental model, subclassers would have access to the name (which they always have, as they can easily spot the mangled name anyways).

If I were reviewing code that uses the __ prefix, I would ask why they're invoking name mangling, and if they couldn't do just as well with a single underscore, keeping in mind that if subclassers choose the same names for the class and class attribute there will be a name collision in spite of this.

Answer 4

"If in doubt about whether a variable should be private or protected, it's better to go with private." - yes, same holds in Python.

Some answers here say about 'conventions', but don't give the links to those conventions. The authoritative guide for Python, PEP 8 states explicitly:

If in doubt, choose non-public; it's easier to make it public later than to make a public attribute non-public.

The distinction between public and private, and name mangling in Python have been considered in other answers. From the same link,

We don't use the term "private" here, since no attribute is really private in Python (without a generally unnecessary amount of work).

Answer 5

There are already a lot of good answers to this, but I'm going to offer another one. This is also partially a response to people who keep saying that double underscore isn't private (it really is).

If you look at Java/C#, both of them have private/protected/public. All of these are compile-time constructs. They are only enforced at the time of compilation. If you were to use reflection in Java/C#, you could easily access private method.

Now every time you call a function in Python, you are inherently using reflection. These pieces of code are the same in Python.

lst = []
lst.append(1)
getattr(lst, 'append')(1)

The "dot" syntax is only syntactic sugar for the latter piece of code. Mostly because using getattr is already ugly with only one function call. It just gets worse from there.

So with that, there can't be a Java/C# version of private, as Python doesn't compile the code. Java and C# can't check if a function is private or public at runtime, as that information is gone (and it has no knowledge of where the function is being called from).

Now with that information, the name mangling of the double underscore makes the most sense for achieving "private-ness". Now when a function is called from the 'self' instance and it notices that it starts with '__', it just performs the name mangling right there. It's just more syntactic sugar. That syntactic sugar allows the equivalent of 'private' in a language that only uses reflection for data member access.

Disclaimer: I have never heard anybody from the Python development say anything like this. The real reason for the lack of "private" is cultural, but you'll also notice that most scripting/interpreted languages have no private. A strictly enforceable private is not practical at anything except for compile time.

Answer 6

#EXAMPLE PROGRAM FOR Python name mangling

class Demo:
    __any_name = "__any_name"
    __any_other_name_ = "__any_other_name_"


[n for n in dir(Demo) if 'any' in n]   # GIVES OUTPUT AS ['_Demo__any_name', 
                                       #    '_Demo__any_other_name_']