Merging duplicates to list of lists

When working with lists of lists in Python, you often encounter duplicate sublists that need to be removed. This article demonstrates two effective approaches to merge (remove) duplicate sublists while preserving unique entries.

Using a For Loop

The simplest approach uses a for loop to iterate through each sublist and check if it already exists in the result ?

def merge_dups(input_list):
    output_list = []
    for each_sublist in input_list:
        if each_sublist not in output_list:
            output_list.append(each_sublist)
    return output_list

my_list = [[3, 10], [8, 2], [3, 10], [15, 6], [3, 17], [7, 8], [3, 10], [15, 6]]
result = merge_dups(my_list)
print("Original:", my_list)
print("After removing duplicates:", result)

Original: [[3, 10], [8, 2], [3, 10], [15, 6], [3, 17], [7, 8], [3, 10], [15, 6]]
After removing duplicates: [[3, 10], [8, 2], [15, 6], [3, 17], [7, 8]]

How It Works

This method maintains the original order of sublists. It creates an empty output list and checks each sublist against existing entries. The first occurrence of each unique sublist is preserved.

Using Set with Tuple Conversion

A more efficient approach converts sublists to tuples, uses a set to remove duplicates, then converts back to lists ?

def remove_duplicates(input_list):
    # Convert sublists to tuples (hashable for set)
    tuple_list = map(tuple, input_list)
    # Use set to eliminate duplicates
    unique_tuple_set = set(tuple_list)
    # Convert back to list of lists
    unique_list = [list(t) for t in unique_tuple_set]
    return unique_list

my_list = [[3, 10], [8, 2], [3, 10], [15, 6], [3, 17], [7, 8], [3, 10], [15, 6]]
result = remove_duplicates(my_list)
print("Original:", my_list)
print("After removing duplicates:", result)

Original: [[3, 10], [8, 2], [3, 10], [15, 6], [3, 17], [7, 8], [3, 10], [15, 6]]
After removing duplicates: [[3, 17], [3, 10], [15, 6], [8, 2], [7, 8]]

Why Convert to Tuples?

Lists are mutable and cannot be added to a set. Tuples are immutable and hashable, making them suitable for set operations. This conversion enables efficient duplicate removal.

Comparison

One-Line Solution

For a more concise approach, you can combine everything into a single line ?

my_list = [[3, 10], [8, 2], [3, 10], [15, 6], [3, 17], [7, 8], [3, 10], [15, 6]]

# One-line solution using set and map
unique_sublists = [list(t) for t in set(map(tuple, my_list))]
print("One-line result:", unique_sublists)

One-line result: [[3, 17], [3, 10], [15, 6], [8, 2], [7, 8]]

Conclusion

Use the for loop method when order preservation is important. Choose the set-based approach for better performance with large datasets. Both methods effectively remove duplicate sublists while maintaining data integrity.