Python - Concatenate Pandas DataFrames Without Duplicates

To concatenate DataFrames, use the concat() method, but to ignore duplicates, use the drop_duplicates() method.

Import the required library −

import pandas as pd

Create DataFrames to be concatenated −

# Create DataFrame1
dataFrame1 = pd.DataFrame(
   {
      "Car": ['BMW', 'Jaguar', 'Audi', 'Mustang'],"Units": [100, 150, 110, 80]
   }
)

# Create DataFrame2
dataFrame2 = pd.DataFrame(
   {
      "Car": ['Tesla', 'Jaguar', 'Mercedes', 'Mustang'],"Units": [120, 150, 180, 80]

   }
)

Now, let us concatenate DataFrame and remove duplicates −

concatRes = pd.concat([dataFrame1, dataFrame2]).drop_duplicates()

Example

Following is the code −

import pandas as pd

# Create DataFrame1
dataFrame1 = pd.DataFrame(
   {
      "Car": ['BMW', 'Jaguar', 'Audi', 'Mustang'],"Units": [100, 150, 110, 80]
   }
)

print"DataFrame1 ...\n",dataFrame1

# Create DataFrame2
dataFrame2 = pd.DataFrame(
   {
      "Car": ['Tesla', 'Jaguar', 'Mercedes', 'Mustang'],"Units": [120, 150, 180, 80]

   }
)

print"\nDataFrame2 ...\n",dataFrame2

# concat DataFrames and remove duplicates
concatRes = pd.concat([dataFrame1, dataFrame2]).drop_duplicates()
print"\nConcatenating DataFrame without duplicates...\n", concatRes

Output

This will produce the following output −

DataFrame1 ...
       Car   Units
0      BMW     100
1   Jaguar     150
2     Audi     110
3  Mustang      80

DataFrame2 ...
        Car   Units
0     Tesla     120
1    Jaguar     150
2  Mercedes     180
3   Mustang      80

Concatenating DataFrame without duplicates...
        Car   Units
0       BMW     100
1    Jaguar     150
2      Audi     110
3   Mustang      80
0     Tesla     120
2  Mercedes     180

AmitDiwan

Updated on: 21-Sep-2021

7K+ Views

Kickstart Your Career

Get certified by completing the course

Get Started