Python - Concatenate Pandas DataFrames Without Duplicates

PythonServer Side ProgrammingProgramming

To concatenate DataFrames, use the concat() method, but to ignore duplicates, use the drop_duplicates() method.

Import the required library −

import pandas as pd

Create DataFrames to be concatenated −

# Create DataFrame1
dataFrame1 = pd.DataFrame(
   {
      "Car": ['BMW', 'Jaguar', 'Audi', 'Mustang'],"Units": [100, 150, 110, 80]
   }
)

# Create DataFrame2
dataFrame2 = pd.DataFrame(
   {
      "Car": ['Tesla', 'Jaguar', 'Mercedes', 'Mustang'],"Units": [120, 150, 180, 80]

   }
)

Now, let us concatenate DataFrame and remove duplicates −

concatRes = pd.concat([dataFrame1, dataFrame2]).drop_duplicates()

Example

Following is the code −

import pandas as pd

# Create DataFrame1
dataFrame1 = pd.DataFrame(
   {
      "Car": ['BMW', 'Jaguar', 'Audi', 'Mustang'],"Units": [100, 150, 110, 80]
   }
)

print"DataFrame1 ...
",dataFrame1 # Create DataFrame2 dataFrame2 = pd.DataFrame(    {       "Car": ['Tesla', 'Jaguar', 'Mercedes', 'Mustang'],"Units": [120, 150, 180, 80] } ) print"
DataFrame2 ...
",dataFrame2 # concat DataFrames and remove duplicates concatRes = pd.concat([dataFrame1, dataFrame2]).drop_duplicates() print"
Concatenating DataFrame without duplicates...
", concatRes

Output

This will produce the following output −

DataFrame1 ...
       Car   Units
0      BMW     100
1   Jaguar     150
2     Audi     110
3  Mustang      80

DataFrame2 ...
        Car   Units
0     Tesla     120
1    Jaguar     150
2  Mercedes     180
3   Mustang      80

Concatenating DataFrame without duplicates...
        Car   Units
0       BMW     100
1    Jaguar     150
2      Audi     110
3   Mustang      80
0     Tesla     120
2  Mercedes     180
raja
Updated on 21-Sep-2021 07:09:41

Advertisements