Get unique values from a list in python [duplicate]

Answer 1

For long arrays

s = np.empty(len(var))

s[:] = np.nan

for  x in  set(var):

    x_positions = np.where(var==x)

    s[x_positions[0][0]]=x


sorted_var=s[~np.isnan(s)]

Answer 2

A Python list:

>>> a = ['a', 'b', 'c', 'd', 'b']

To get unique items, just transform it into a set (which you can transform back again into a list if required):

>>> b = set(a)
>>> print(b)
{'b', 'c', 'd', 'a'}

Answer 3

Maintaining order:

# oneliners
# slow -> . --- 14.417 seconds ---
[x for i, x in enumerate(array) if x not in array[0:i]]

# fast -> . --- 0.0378 seconds ---
[x for i, x in enumerate(array) if array.index(x) == i]

# multiple lines
# fastest -> --- 0.012 seconds ---
uniq = []
[uniq.append(x) for x in array if x not in uniq]
uniq

Order doesn't matter:

# fastest-est -> --- 0.0035 seconds ---
list(set(array))

Answer 4

I am surprised that nobody so far has given a direct order-preserving answer:

def unique(sequence):
    """Generate unique items from sequence in the order of first occurrence."""
    seen = set()
    for value in sequence:
        if value in seen:
            continue

        seen.add(value)

        yield value

It will generate the values so it works with more than just lists, e.g. unique(range(10)). To get a list, just call list(unique(sequence)), like this:

>>> list(unique([u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']))
[u'nowplaying', u'PBS', u'job', u'debate', u'thenandnow']

It has the requirement that each item is hashable and not just comparable, but most stuff in Python is and it is O(n) and not O(n^2), so will work just fine with a long list.

Answer 5

I know this is an old question, but here's my unique solution: class inheritance!:

class UniqueList(list):
    def appendunique(self,item):
        if item not in self:
            self.append(item)
            return True
        return False

Then, if you want to uniquely append items to a list you just call appendunique on a UniqueList. Because it inherits from a list, it basically acts like a list, so you can use functions like index() etc. And because it returns true or false, you can find out if appending succeeded (unique item) or failed (already in the list).

To get a unique list of items from a list, use a for loop appending items to a UniqueList (then copy over to the list).

Example usage code:

unique = UniqueList()

for each in [1,2,2,3,3,4]:
    if unique.appendunique(each):
        print 'Uniquely appended ' + str(each)
    else:
        print 'Already contains ' + str(each)

Prints:

Uniquely appended 1
Uniquely appended 2
Already contains 2
Uniquely appended 3
Already contains 3
Uniquely appended 4

Copying to list:

unique = UniqueList()

for each in [1,2,2,3,3,4]:
    unique.appendunique(each)

newlist = unique[:]
print newlist

Prints:

[1, 2, 3, 4]

Answer 6

Getting unique elements from List

mylist = [1,2,3,4,5,6,6,7,7,8,8,9,9,10]

Using Simple Logic from Sets - Sets are unique list of items

mylist=list(set(mylist))

In [0]: mylist
Out[0]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Using Simple Logic

newList=[]
for i in mylist:
    if i not in newList:
        newList.append(i)

In [0]: mylist
Out[0]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Using pop method ->pop removes the last or indexed item and displays that to user. video

k=0
while k < len(mylist):
    if mylist[k] in mylist[k+1:]:
        mylist.pop(mylist[k])
    else:
        k=k+1

In [0]: mylist
Out[0]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Using Numpy

import numpy as np
np.unique(mylist)

In [0]: mylist
Out[0]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Reference