How to find the common elements in a Pandas DataFrame?

Finding common elements between Pandas DataFrames is essential for data analysis tasks like identifying shared records or performing data validation. Python provides several methods including merge(), intersection(), and set operations.

Using merge() Method

The merge() method performs an inner join by default, returning only common rows between DataFrames −

import pandas as pd

df1 = pd.DataFrame({
    "x": [5, 2, 7, 0],
    "y": [4, 7, 5, 1],
    "z": [9, 3, 5, 1]
})

df2 = pd.DataFrame({
    "x": [5, 2, 7, 0, 11, 12],
    "y": [4, 7, 5, 1, 19, 20],
    "z": [9, 3, 5, 1, 29, 30]
})

print("DataFrame 1:")
print(df1)
print("\nDataFrame 2:")
print(df2)

# Find common rows
common = df1.merge(df2, on=['x', 'y', 'z'])
print("\nCommon rows:")
print(common)

DataFrame 1:
   x  y  z
0  5  4  9
1  2  7  3
2  7  5  5
3  0  1  1

DataFrame 2:
   x  y  z
0   5  4  9
1   2  7  3
2   7  5  5
3   0  1  1
4  11 19 29
5  12 20 30

Common rows:
   x  y  z
0  5  4  9
1  2  7  3
2  7  5  5
3  0  1  1

Using intersection() for Column Names

To find common column names between DataFrames −

import pandas as pd

df1 = pd.DataFrame({
    "name": ["Alice", "Bob"],
    "age": [25, 30],
    "city": ["NY", "LA"]
})

df2 = pd.DataFrame({
    "name": ["Charlie", "David"],
    "age": [35, 40],
    "country": ["USA", "Canada"]
})

# Find common column names
common_columns = df1.columns.intersection(df2.columns)
print("Common columns:", common_columns.tolist())

Common columns: ['name', 'age']

Using Set Operations for Index Values

To find common index values between DataFrames −

import pandas as pd

df1 = pd.DataFrame({
    "value": [10, 20, 30]
}, index=[1, 2, 3])

df2 = pd.DataFrame({
    "value": [40, 50, 60]
}, index=[2, 3, 4])

# Find common index values
common_indices = set(df1.index) & set(df2.index)
print("Common indices:", sorted(common_indices))

Common indices: [2, 3]

Comparison

Conclusion

Use merge() to find common rows between DataFrames. Use intersection() for common column names and set operations for common index values.