How do you remove duplicates from a list in Python?

Removing duplicates from a list is a common task in Python. There are several efficient approaches: using set(), OrderedDict, and list comprehension. Each method has different characteristics regarding order preservation and performance.

Using set() Method

The simplest approach converts the list to a set, which automatically removes duplicates. However, this doesn't preserve the original order ?

# Creating a List with duplicate items
names = ["Jacob", "Harry", "Mark", "Anthony", "Harry", "Anthony"]

# Displaying the original List
print("Original List =", names)

# Remove duplicates using set
unique_names = list(set(names))
print("Updated List =", unique_names)

Original List = ['Jacob', 'Harry', 'Mark', 'Anthony', 'Harry', 'Anthony']
Updated List = ['Anthony', 'Mark', 'Jacob', 'Harry']

Using OrderedDict (Order Preserved)

To maintain the original order while removing duplicates, use OrderedDict.fromkeys() ?

from collections import OrderedDict

# Creating a List with duplicate items
names = ["Jacob", "Harry", "Mark", "Anthony", "Harry", "Anthony"]

# Displaying the original List
print("Original List =", names)

# Remove duplicates using OrderedDict while preserving order
unique_names = list(OrderedDict.fromkeys(names))
print("Updated List =", unique_names)

Original List = ['Jacob', 'Harry', 'Mark', 'Anthony', 'Harry', 'Anthony']
Updated List = ['Jacob', 'Harry', 'Mark', 'Anthony']

Using List Comprehension

List comprehension with a tracking list preserves order but is less efficient for large lists ?

# Creating a List with duplicate items
names = ["Jacob", "Harry", "Mark", "Anthony", "Harry", "Anthony"]

# Displaying the original List
print("Original List =", names)

# Remove duplicates using list comprehension
unique_names = []
[unique_names.append(item) for item in names if item not in unique_names]
print("Updated List =", unique_names)

Original List = ['Jacob', 'Harry', 'Mark', 'Anthony', 'Harry', 'Anthony']
Updated List = ['Jacob', 'Harry', 'Mark', 'Anthony']

Using dict.fromkeys() (Python 3.7+)

A modern approach that preserves order and is more efficient than OrderedDict ?

# Creating a List with duplicate items
names = ["Jacob", "Harry", "Mark", "Anthony", "Harry", "Anthony"]

# Displaying the original List
print("Original List =", names)

# Remove duplicates using dict.fromkeys()
unique_names = list(dict.fromkeys(names))
print("Updated List =", unique_names)

Original List = ['Jacob', 'Harry', 'Mark', 'Anthony', 'Harry', 'Anthony']
Updated List = ['Jacob', 'Harry', 'Mark', 'Anthony']

Comparison

Conclusion

Use set() for fastest duplicate removal when order doesn't matter. Use dict.fromkeys() for order-preserving removal in modern Python. Use OrderedDict for older Python versions that need order preservation.