Programming Tutorials

Remove duplicates from a list in python

By: Python Documentation Team  

If you don’t mind reordering the list, sort it and then scan from the end of the list, deleting duplicates as you go:

if mylist:
    mylist.sort()
    last = mylist[-1]
    for i in range(len(mylist)-2, -1, -1):
        if last == mylist[i]:
            del mylist[i]
        else:
            last = mylist[i]
If all elements of the list may be used as dictionary keys (i.e. they are all hashable) this is often faster

d = {}
for x in mylist:
    d[x] = 1
mylist = list(d.keys())
In Python 2.5 and later, the following is possible instead:

mylist = list(set(mylist))
This converts the list into a set, thereby removing duplicates, and then back into a list.

You can also see the Python Cookbook for a long discussion of many ways to do this:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560