Get unique values from a list in python [duplicate]

Answer 1

If you are using numpy in your code (which might be a good choice for larger amounts of data), check out numpy.unique:

>>> import numpy as np
>>> wordsList = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
>>> np.unique(wordsList)
array([u'PBS', u'debate', u'job', u'nowplaying', u'thenandnow'], 
      dtype='<U10')

(http://docs.scipy.org/doc/numpy/reference/generated/numpy.unique.html)

As you can see, numpy supports not only numeric data, string arrays are also possible. Of course, the result is a numpy array, but it doesn't matter a lot, because it still behaves like a sequence:

>>> for word in np.unique(wordsList):
...     print word
... 
PBS
debate
job
nowplaying
thenandnow

If you really want to have a vanilla python list back, you can always call list().

However, the result is automatically sorted, as you can see from the above code fragments. Check out numpy unique without sort if retaining list order is required.

Answer 2

set - unordered collection of unique elements. List of elements can be passed to set's constructor. So, pass list with duplicate elements, we get set with unique elements and transform it back to list then get list with unique elements. I can say nothing about performance and memory overhead, but I hope, it's not so important with small lists.

list(set(my_not_unique_list))

Simply and short.

Answer 3

Same order unique list using only a list compression.

> my_list = [1, 2, 1, 3, 2, 4, 3, 5, 4, 3, 2, 3, 1]
> unique_list = [
>    e
>    for i, e in enumerate(my_list)
>    if my_list.index(e) == i
> ]
> unique_list
[1, 2, 3, 4, 5]

enumerates gives the index i and element e as a tuple.

my_list.index returns the first index of e. If the first index isn't i then the current iteration's e is not the first e in the list.

Edit

I should note that this isn't a good way to do it, performance-wise. This is just a way that achieves it using only a list compression.

Answer 4

First thing, the example you gave is not a valid list.

example_list = [u'nowplaying',u'PBS', u'PBS', u'nowplaying', u'job', u'debate',u'thenandnow']

Suppose if above is the example list. Then you can use the following recipe as give the itertools example doc that can return the unique values and preserving the order as you seem to require. The iterable here is the example_list

from itertools import ifilterfalse

def unique_everseen(iterable, key=None):
    "List unique elements, preserving order. Remember all elements ever seen."
    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
    # unique_everseen('ABBCcAD', str.lower) --> A B C D
    seen = set()
    seen_add = seen.add
    if key is None:
        for element in ifilterfalse(seen.__contains__, iterable):
            seen_add(element)
            yield element
    else:
        for element in iterable:
            k = key(element)
            if k not in seen:
                seen_add(k)
                yield element

Answer 5

Use the following function:

def uniquefy_list(input_list):
"""
This function  takes a list as input and return a list containing only unique elements from the input list

"""
output_list=[]
for elm123 in input_list:
    in_both_lists=0
    for elm234 in output_list:
        if elm123 == elm234:
            in_both_lists=1
            break
    if in_both_lists == 0:
        output_list.append(elm123)

return output_list

Answer 6

def setlist(lst=[]):
   return list(set(lst))