Why does removing duplicates from a list produce [None, None] output?

Question

I am new to Python and I\'m not able to understand why I am getting the results with None values.

#Remove duplicate items from a list
def remove         


        

Answer 1

Although using a set is the proper way, the problem with your code, as the comments indicated, is that you are not actually returning unique_list from your function, you are returning the result of the list comprehension.

def remove_duplicates(my_list):
    unique_list = []
    do = [unique_list.append(item) for item in my_list if item not in unique_list]
    return unique_list  # Actually return the list!

print remove_duplicates([1,1,2,2]) -> result [1, 2]

Here I simply made a throwaway variable do that is useless, it just "runs" the comprehension. Understand?

That comprehension is storing a value each time you call unique_list.append(item) ... and that value is the result of the append method, which is None! So do equals [None, None].

However, your unique_list is in fact being populated correctly, so we can return that and now your function works as expected.

Of course, this is not a normal use for a list comprehension and really weird.

Answer 2

As the other answers have explained, the reason you're getting a list of None values is because list.append returns None, and you're calling it in a list comprehension. That means you're building a list full of None values along side your list of unique values.

I would like to suggest that you ditch the list comprehension. Because you need to access outside state (the list of unique values seen so far), a comprehension can't easily do what you want. A regular for loop is much more appropriate:

def remove_duplicates(lst):
    unique_list = []
    for item in lst:
        if item not in unique_list:
            unique_list.append(item)
    return unique_list

A more Pythonic approach however would be to use a set to handle the unique items, and to make your function a generator:

def remove_duplicates(lst):
    uniques = set()
    for item in lst:
        if item not in unique_list:
            yield item
            uniques.add(item)

The itertools.ifilterfase function from the standard library can help improve this even further, as shown in the recipe in the docs (you'll have to scroll down a little to find the specific recipe):

def unique_everseen(iterable, key=None):
    "List unique elements, preserving order. Remember all elements ever seen."
    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
    # unique_everseen('ABBCcAD', str.lower) --> A B C D
    seen = set()
    seen_add = seen.add
    if key is None:
        for element in filterfalse(seen.__contains__, iterable):
            seen_add(element)
            yield element
    else:
        for element in iterable:
            k = key(element)
            if k not in seen:
                seen_add(k)
                yield element

Answer 3

The problem with your code is that the method list.append returns None. You can test this easily with the following code:

myList=[1, 2, 3]
print myList.append(4)

So, a solution for you would issue would be

def remove_duplicates(myList):
    alreadyIncluded = []
    return [item for item in myList if item not in alreadyIncluded and not alreadyIncluded.append(item)]
print remove_duplicates([1,1,2,2]) 

The idea is that you will begin with an empty list of aldeady included elements and you will loop over all the elements in list, including them in the alreadyIncluded list. The not is necessary because the append will return None and not None is True, so the if will not be affected by the inclusion.

You were including a list of the result of the appends (always None), but what you need is a list of the elements that passed the if test.

I hope it helps.