Duplicate substring removal from list in Python

Sometimes we may have a need to refine a given list by eliminating the duplicate elements in it. This can be achieved by using a combination of various methods available in python standard library.

with set and split

The split method can be used to segregate the elements for duplicate checking and the set method is used to store the unique elements from the segregated list elements.

Example

# initializing list
listA = [ 'xy-xy', 'pq-qr', 'xp-xp-xp', 'dd-ee']

print("Given list : ",listA)

# using set() and split()
res = [set(sub.split('-')) for sub in listA]

# Result
print("List after duplicate removal : " ,res)

Output

Running the above code gives us the following result −

Given list : ['xy-xy', 'pq-qr', 'xp-xp-xp', 'dd-ee']
List after duplicate removal : [{'xy'}, {'pq', 'qr'}, {'xp'}, {'ee', 'dd'}]

With list

We can also use the list method and use for loops along with it so that only unique elements from the list after segregations are captured.

Example

# initializing list
listA = [ 'xy-xy', 'pq-qr', 'xp-xp-xp', 'dd-ee']

print("Given list : ",listA)

# using list
res = list({i for sub in listA for i in sub.split('-')})

# Result
print("List after duplicate removal : " , res)

Output

Running the above code gives us the following result −

Given list : ['xy-xy', 'pq-qr', 'xp-xp-xp', 'dd-ee']
List after duplicate removal : ['dd', 'pq', 'ee', 'xp', 'xy', 'qr']