Remove all duplicates and permutations in a nested list in Python

Removing duplicates and permutations from a nested list in Python is a common task that helps streamline data and avoid redundant or repetitive elements. In this article, we aim to extract a unique set of sublists from the nested list, eliminating any duplicates or permutations. By doing so, we will streamline further operations and ensure data integrity. We will explore three different approaches to achieve this objective with step-by-step explanations, Python code, and output.

Approach 1: Flattening and Converting to Set

This approach involves flattening the nested list into a single list and then converting it to a set to remove duplicates. Here are the steps to implement this approach:

Algorithm

• Step 1 Create a function called flatten_list that takes a nested list as input.

• Step 2 Initialize an empty list, flattened_list, to store the flattened elements.

• Step 3 Iterate through each element in the nested list.

• a. If the element is a list, recursively call the flatten_list function on it.

• b. If the element is not a list, add it to the flattened_list.

• Step 4 Return the flattened_list.

• Step 5 After getting the flattened list, convert it to a set using the set() function.

• Step 6 Convert the set back to a list to preserve unique elements.

Example

The following example demonstrates flattening a nested list and removing duplicates ?

def flatten_list(nested_list):
    flattened_list = []
    
    for element in nested_list:
        if isinstance(element, list):
            flattened_list.extend(flatten_list(element))
        else:
            flattened_list.append(element)
    
    return flattened_list

nested_list = [13, 2, [3, 45, [5, 56]], [7, 68, [9, 10]]]
flattened_list = flatten_list(nested_list)
result = list(set(flattened_list))

print("Original nested list:", nested_list)
print("Flattened unique elements:", result)

Original nested list: [13, 2, [3, 45, [5, 56]], [7, 68, [9, 10]]]
Flattened unique elements: [2, 3, 68, 5, 7, 9, 10, 45, 13, 56]

Approach 2: Removing Duplicates within Sublists

In this approach, we define a function that removes duplicate elements within each sublist of the nested list. This approach preserves the nested structure while ensuring each sublist contains only unique elements ?

Algorithm

• Step 1 Define a helper function remove_duplicates_helper() that takes a sublist as input.

• Step 2 Sort the sublist to group duplicate elements together.

• Step 3 Use list comprehension to keep only unique elements.

• Step 4 Apply this helper function to each sublist in the nested list.

Example

The following example removes duplicates within each sublist ?

def remove_duplicates_helper(sublist):
    sublist.sort()
    unique_sublist = [sublist[i] for i in range(len(sublist)) if i == 0 or sublist[i] != sublist[i - 1]]
    return unique_sublist

def remove_duplicates_in_sublists(nested_list):
    result = []
    for sublist in nested_list:
        result.append(remove_duplicates_helper(sublist.copy()))
    return result

# Example usage
nested_list = [[1, 25, 1], [42, 3, 44, 4, 3], [5, 6, 76, 5]]
result = remove_duplicates_in_sublists(nested_list)

print("Original nested list:", nested_list)
print("After removing duplicates:", result)

Original nested list: [[1, 25, 1], [42, 3, 44, 4, 3], [5, 6, 76, 5]]
After removing duplicates: [[1, 25], [3, 4, 42, 44], [5, 6, 76]]

Approach 3: Removing Duplicate Sublists

In this approach, we iterate over each sublist in the nested list and compare it with subsequent sublists to remove exact duplicates ?

Algorithm

• Step 1 Define a function named remove_duplicate_sublists().

• Step 2 Iterate over each sublist in the nested list.

• Step 3 Compare each sublist with the subsequent sublists.

• Step 4 Remove any duplicate sublists found.

• Step 5 Return the modified nested list.

Example

The following example removes duplicate sublists from the nested list ?

def remove_duplicate_sublists(nested_list):
    result = []
    for i in range(len(nested_list)):
        is_duplicate = False
        for j in range(i + 1, len(nested_list)):
            if nested_list[i] == nested_list[j]:
                is_duplicate = True
                break
        if not is_duplicate:
            result.append(nested_list[i])
    return result

# Example usage
nested_list = [[1, 25, 13], [32, 3, 64], [1, 25, 13], [5, 6, 7]]
result = remove_duplicate_sublists(nested_list)

print("Original nested list:", nested_list)
print("After removing duplicate sublists:", result)

Original nested list: [[1, 25, 13], [32, 3, 64], [1, 25, 13], [5, 6, 7]]
After removing duplicate sublists: [[1, 25, 13], [32, 3, 64], [5, 6, 7]]

Comparison

Conclusion

We explored three different approaches to remove duplicates and permutations from a nested list in Python. Use flattening for a single unique list, sublist deduplication to clean individual sublists, or duplicate sublist removal to eliminate repeated sublists entirely.