Python – Remove Rows for similar Kth column element

When working with lists of lists in Python, you may need to remove rows that have duplicate values in a specific column (Kth position). This can be achieved using simple iteration and the append() method to build a new list with unique values.

Example

Below is a demonstration of removing rows with duplicate Kth column elements ?

data = [[45, 95, 26], [70, 35, 74], [87, 65, 23], [70, 35, 74], [67, 85, 12], [45, 65, 0]]

print("The list is:")
print(data)

K = 1
print("The value of K is:")
print(K)

result = []
seen_values = []

for row in data:
    if row[K] not in seen_values:
        result.append(row)
        seen_values.append(row[K])

print("The resultant list is:")
print(result)

Output

The list is:
[[45, 95, 26], [70, 35, 74], [87, 65, 23], [70, 35, 74], [67, 85, 12], [45, 65, 0]]
The value of K is:
1
The resultant list is:
[[45, 95, 26], [70, 35, 74], [87, 65, 23], [67, 85, 12]]

How It Works

The algorithm maintains two lists:

• result ? Stores the filtered rows without duplicates
• seen_values ? Tracks the Kth column values already encountered

For each row, it checks if the Kth column value exists in seen_values. If not, the row is added to result and its Kth value is recorded.

Alternative Using Set

For better performance with larger datasets, use a set to track seen values ?

data = [[45, 95, 26], [70, 35, 74], [87, 65, 23], [70, 35, 74], [67, 85, 12]]
K = 1

result = []
seen = set()

for row in data:
    if row[K] not in seen:
        result.append(row)
        seen.add(row[K])

print("Filtered list:", result)

Filtered list: [[45, 95, 26], [70, 35, 74], [87, 65, 23], [67, 85, 12]]

Conclusion

Use iteration with a tracking list to remove rows with duplicate Kth column values. For larger datasets, prefer using a set for O(1) lookup performance instead of a list's O(n) lookup.