Get unique values from a list in python [duplicate]

Answer 1

Options to remove duplicates may include the following generic data structures:

• set: unordered, unique elements
• ordered set: ordered, unique elements

Here is a summary on quickly getting either one in Python.

Given

from collections import OrderedDict


seq = [u"nowplaying", u"PBS", u"PBS", u"nowplaying", u"job", u"debate", u"thenandnow"]

Code

Option 1 - A set (unordered):

list(set(seq))
# ['thenandnow', 'PBS', 'debate', 'job', 'nowplaying']

Option 2 - Python doesn't have ordered sets, but here are some ways to mimic one (insertion ordered):

list(OrderedDict.fromkeys(seq))
# ['nowplaying', 'PBS', 'job', 'debate', 'thenandnow']

list(dict.fromkeys(seq))                               # py36
# ['nowplaying', 'PBS', 'job', 'debate', 'thenandnow']

The last option is recommended if using Python 3.6+. See more details in this post.

Note: listed elements must be hashable. See details on the latter example in this blog post. Furthermore, see R. Hettinger's post on the same technique; the order preserving dict is extended from one of his early implementations. See also more on total ordering.

Answer 2

You can use sets. Just to be clear, I am explaining what is the difference between a list and a set. sets are unordered collection of unique elements.Lists are ordered collection of elements. So,

    unicode_list=[u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job',u'debate', u'thenandnow']
    list_unique=list(set(unicode_list))
    print list_unique
[u'nowplaying', u'job', u'debate', u'PBS', u'thenandnow']

But: Do not use list/set in naming the variables. It will cause error: EX: Instead of use list instead of unicode_list in the above one.

list=[u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job',u'debate', u'thenandnow']
        list_unique=list(set(list))
        print list_unique
    list_unique=list(set(list))
TypeError: 'list' object is not callable

Answer 3

What type is your output variable?

Python sets are what you need. Declare output like this:

output = set()  # initialize an empty set

and you're ready to go adding elements with output.add(elem) and be sure they're unique.

Warning: sets DO NOT preserve the original order of the list.

Answer 4

By using basic property of Python Dictionary:

inp=[u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
d={i for i in inp}
print d

Output will be:

set([u'nowplaying', u'job', u'debate', u'PBS', u'thenandnow'])

Answer 5

If you want to get unique elements from a list and keep their original order, then you may employ OrderedDict data structure from Python's standard library:

from collections import OrderedDict

def keep_unique(elements):
    return list(OrderedDict.fromkeys(elements).keys())

elements = [2, 1, 4, 2, 1, 1, 5, 3, 1, 1]
required_output = [2, 1, 4, 5, 3]

assert keep_unique(elements) == required_output

In fact, if you are using Python ≥ 3.6, you can use plain dict for that:

def keep_unique(elements):
    return list(dict.fromkeys(elements).keys())

It's become possible after the introduction of "compact" representation of dicts. Check it out here. Though this "considered an implementation detail and should not be relied upon".

Answer 6

To be consistent with the type I would use:

mylist = list(set(mylist))