How can I remove the same element in the list by Python

A list is a built-in Python data structure that stores an ordered collection of items. Lists often contain duplicate elements, which can cause data inaccuracies. In this article, we will explore several approaches to remove duplicate elements from a list while preserving the original order.

Using set() Function

The set() function accepts an iterable (like a list, tuple, or string) as a parameter and creates a set object. Since a set object does not accept duplicate values, converting a list to a set automatically removes duplicates.

Example

The following example removes duplicate elements using the set() function ?

numbers = [1, 2, 3, 4, 4, 5, 66, 3, 2, 6, 9, 66]

# Remove duplicates by converting to a set
unique_numbers = list(set(numbers))

print(unique_numbers)

The output of the above code is ?

[1, 2, 3, 4, 5, 66, 6, 9]

Note: This method does not preserve the original order of elements.

Using a For Loop

To remove duplicate values while preserving order, we can iterate through the list using a for loop and store unique elements in a new list using the not in operator.

Example

The following example uses a brute force approach to remove duplicate elements ?

numbers = [1, 2, 3, 4, 4, 5, 66, 3, 2, 6, 9, 66]

# Create an empty list to store unique values
result = []

# Iterate through each value in the list
for val in numbers:  
    # Check if the value is not already in result
    if val not in result:      
        # If not present, append it to result
        result.append(val)

print(result)

The output of the above code is ?

[1, 2, 3, 4, 5, 66, 6, 9]

Using List Comprehension

List comprehension provides a concise syntax for creating a new list by performing operations on an existing list. We can combine it with the not in operator to filter unique elements.

Example

The following example uses list comprehension to remove duplicates while preserving order ?

numbers = [1, 2, 3, 4, 4, 5, 66, 3, 2, 6, 9, 66]

# Create an empty list to store unique values
result = []

# Use list comprehension to append unique values
[result.append(val) for val in numbers if val not in result]

print(result)

The output of the above code is ?

[1, 2, 3, 4, 5, 66, 6, 9]

Using Dictionary fromkeys()

A Dictionary in Python stores data in key-value pairs. The fromkeys() method accepts an iterable and creates a dictionary with those values as keys. Since dictionary keys are unique, this method effectively removes duplicates while preserving order.

Example

The following example uses the fromkeys() method to remove duplicates ?

numbers = [1, 2, 3, 4, 4, 5, 66, 3, 2, 6, 9, 66]

# Remove duplicates using dictionary fromkeys()
unique_numbers = list(dict.fromkeys(numbers))

print(unique_numbers)

The output of the above code is ?

[1, 2, 3, 4, 5, 66, 6, 9]

Comparison of Methods

Conclusion

Use dict.fromkeys() for the best balance of performance and order preservation. Use set() when order doesn't matter and maximum speed is needed. The for loop and list comprehension methods are educational but less efficient for large datasets.