Get unique values from a list in python [duplicate]

Answer 1

My solution to check contents for uniqueness but preserve the original order:

def getUnique(self):
    notunique = self.readLines()
    unique = []
    for line in notunique: # Loop over content
        append = True # Will be set to false if line matches existing line
        for existing in unique:
            if line == existing: # Line exists ? do not append and go to the next line
                append = False
                break # Already know file is unique, break loop
        if append: unique.append(line) # Line not found? add to list
    return unique

Edit: Probably can be more efficient by using dictionary keys to check for existence instead of doing a whole file loop for each line, I wouldn't use my solution for large sets.

Answer 2

Try this function, it's similar to your code but it's a dynamic range.

def unique(a):

    k=0
    while k < len(a):
        if a[k] in a[k+1:]:
            a.pop(k)
        else:
            k=k+1



    return a

Answer 3

As a bonus, Counter is a simple way to get both the unique values and the count for each value:

from collections import Counter
l = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
c = Counter(l)

Answer 4

set can help you filter out the elements from the list that are duplicates. It will work well for str, int or tuple elements, but if your list contains dict or other list elements, then you will end up with TypeError exceptions.

Here is a general order-preserving solution to handle some (not all) non-hashable types:

def unique_elements(iterable):
    seen = set()
    result = []
    for element in iterable:
        hashed = element
        if isinstance(element, dict):
            hashed = tuple(sorted(element.iteritems()))
        elif isinstance(element, list):
            hashed = tuple(element)
        if hashed not in seen:
            result.append(element)
            seen.add(hashed)
    return result

Answer 5

use set to de-duplicate a list, return as list

def get_unique_list(lst):
        if isinstance(lst,list):
            return list(set(lst))

Answer 6

To get unique values from your list use code below:

trends = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
output = set(trends)
output = list(output)

IMPORTANT: Approach above won't work if any of items in a list is not hashable which is case for mutable types, for instance list or dict.

trends = [{'super':u'nowplaying'}, u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
output = set(trends)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  TypeError: unhashable type: 'dict'

That means that you have to be sure that trends list would always contains only hashable items otherwise you have to use more sophisticated code:

from copy import deepcopy

try:
    trends = [{'super':u'nowplaying'}, [u'PBS',], [u'PBS',], u'nowplaying', u'job', u'debate', u'thenandnow', {'super':u'nowplaying'}]
    output = set(trends)
    output = list(output)
except TypeError:
    trends_copy = deepcopy(trends)
    while trends_copy:
        trend = trends_copy.pop()
        if trends_copy.count(trend) == 0:
            output.append(trend)
print output