Learn how to remove duplicates from a List in Python.
Example
Remove any duplicates from a List:
mylist = ["a", "b", "a", "c", "c"]
mylist = list[dict.fromkeys[mylist]]
print[mylist]
Try it Yourself »
Example Explained
First we have a List that contains duplicates:
A List with Duplicates
mylist = ["a", "b", "a", "c", "c"]
mylist = list[dict.fromkeys[mylist]]
print[mylist]
Create a dictionary, using the List items as keys. This will automatically remove any duplicates because dictionaries cannot have duplicate keys.
Create a Dictionary
mylist = ["a", "b", "a", "c", "c"]
mylist = list[dict.fromkeys[mylist]]
print[mylist]
Then, convert the dictionary back into a list:
Convert Into a List
mylist = ["a", "b", "a", "c", "c"]
mylist = list[dict.fromkeys[mylist]]
print[mylist]
Now we have a List without any duplicates, and it has the same order as the original List.
Print the List to demonstrate the result
Print the List
mylist = ["a", "b", "a", "c", "c"]
mylist
= list[dict.fromkeys[mylist]]
print[mylist]
Create a Function
If you like to have a function where you can send your lists, and get them back without duplicates, you can create a function and insert the code from the example above.
Example
def my_function[x]:
return list[dict.fromkeys[x]]
mylist = my_function[["a", "b", "a", "c", "c"]]
print[mylist]
Try it Yourself »
Example Explained
Create a function that takes a List as an argument.
Create a Function
def my_function[x]:
return list[dict.fromkeys[x]]
mylist = my_function[["a", "b", "a", "c", "c"]]
print[mylist]
Create a dictionary, using this List items as keys.
Create a Dictionary
def my_function[x]:
return list[dict.fromkeys[x]]
mylist = my_function[["a", "b", "a", "c", "c"]]
print[mylist]
Convert the dictionary into a list.
Convert Into a List
def my_function[x]:
return list[dict.fromkeys[x]]
mylist = my_function[["a", "b", "a", "c", "c"]]
print[mylist]
Return the list
Return List
def my_function[x]:
return list[dict.fromkeys[x]]
mylist = my_function[["a", "b", "a", "c", "c"]]
print[mylist]
Call the function, with a list as a parameter:
Call the Function
def my_function[x]:
return list[dict.fromkeys[x]]mylist = my_function[["a", "b", "a", "c",
"c"]]print[mylist]
Print the result:
Print the Result
def my_function[x]:
return list[dict.fromkeys[x]]
mylist = my_function[["a", "b", "a", "c", "c"]]
print[mylist]This article focuses on one of the operations of getting the unique list from a list that contains a possible duplicated. Remove duplicates from list operation has a large number of applications and hence, its knowledge is good to have.
Method 1: Using *set[]
This is the fastest and smallest method to achieve a particular task. It first removes the duplicates and returns a dictionary which has to be converted to list.
Python3
l
=
[
1
,
2
,
4
,
2
,
1
,
4
,
5
]
print
[
"Original List: "
, l]
res
=
[
*
set
[l]]
print
[
"List after removing duplicate elements: "
, res]
Output:
Original list: [1, 2, 4, 2, 1, 4, 5] List after removing duplicate elements: [1, 2, 4, 5]
Method 2: Using list comprehension
This method has working similarly to the above method, but this is just a one-liner shorthand of a longer method done with the help of list comprehension.
Python3
test_list
=
[
1
,
3
,
5
,
6
,
3
,
5
,
6
,
1
]
print
[& quot
The original
list
is
: & quot
+
str
[test_list]]
res
=
[]
[res.append[x]
for
x
in
test_list
if
x
not
in
res]
print
[& quot
The
list
after removing duplicates : & quot
+
str
[res]]
Output :
The original list is : [1, 3, 5, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 3, 5, 6]
Method 3: Using set[]
This is the most popular way by which the duplicates are removed from the list. But the main and notable drawback of this approach is that the ordering of the element is lost in this particular method.
Python3
test_list
=
[
1
,
5
,
3
,
6
,
3
,
5
,
6
,
1
]
print
[& quot
The original
list
is
: & quot
+
str
[test_list]]
test_list
=
list
[
set
[test_list]]
print
[& quot
The
list
after removing duplicates : & quot
+
str
[test_list]]
Output :
The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 3, 5, 6]
Method 4: Using list comprehension + enumerate[]
list comprehension coupled with enumerate function can also achieve this task. It basically looks for already occurred elements and skips adding them. It preserves the list ordering.
Python3
test_list
=
[
1
,
5
,
3
,
6
,
3
,
5
,
6
,
1
]
print
[& quot
The original
list
is
: & quot
+
str
[test_list]]
res
=
[i
for
n, i
in
enumerate
[test_list]
if
i
not
in
test_list[:n]]
print
[& quot
The
list
after removing duplicates : & quot
+
str
[res]]
Output :
The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 5, 3, 6]
Method 5: Using collections.OrderedDict.fromkeys[]
This is the fastest method to achieve a particular task. It first removes the duplicates and returns a dictionary which has to be converted to list. This works well in the case of strings also.
Python3
from
collections
import
OrderedDict
test_list
=
[
1
,
5
,
3
,
6
,
3
,
5
,
6
,
1
]
print
[ & quot
The original
list
is
: & quot
+
str
[test_list]]
res
=
list
[OrderedDict.fromkeys[test_list]]
print
[ & quot
The
list
after removing duplicates : & quot
+
str
[res]]
Output:
The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 5, 3, 6]
Method 6: Using in, not in operators
Python3
test_list
=
[
1
,
5
,
3
,
6
,
3
,
5
,
6
,
1
]
print
[
"The original list is : "
+
str
[test_list]]
res
=
[]
for
i
in
test_list:
if
i
not
in
res:
res.append[i]
print
[
"The list after removing duplicates : "
+
str
[res]]
Output
The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 5, 3, 6]
Method 7: Using list comprehension and Array.index[] method.
In this method we use list comprehension to iterate over the list and array indexing to get the item from array. We add item to array only if first index of element in array match with current index of element else neglect the element.
Python
arr
=
[
1
,
5
,
3
,
6
,
3
,
5
,
6
,
1
]
print
[
'The original list is : '
+
str
[arr]]
res
=
[arr[i]
for
i
in
range
[
len
[arr]]
if
i
=
=
arr.index[arr[i]] ]
print
[
'The list after removing duplicates :'
,res]
Output:
The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 5, 3, 6]