Python - Remove duplicate values from a Pandas DataFrame

To remove duplicate values from a Pandas DataFrame, use the drop_duplicates() method. At first, create a DataFrame with 3 columns −

dataFrame = pd.DataFrame({'Car': ['BMW', 'Mercedes', 'Lamborghini', 'BMW', 'Mercedes', 'Porsche'],'Place': ['Delhi', 'Hyderabad', 'Chandigarh', 'Delhi', 'Hyderabad', 'Mumbai'],'UnitsSold': [95, 70, 80, 95, 70, 90]})

Remove duplicate values −

dataFrame = dataFrame.drop_duplicates()

Example

Following is the complete code −

import pandas as pd

# Create DataFrame
dataFrame = pd.DataFrame({'Car': ['BMW', 'Mercedes', 'Lamborghini', 'BMW', 'Mercedes', 'Porsche'],'Place': ['Delhi', 'Hyderabad', 'Chandigarh', 'Delhi', 'Hyderabad', 'Mumbai'], 'UnitsSold': [95, 70, 80, 95, 70, 90]})

print"Dataframe...\n", dataFrame

# counting frequency of column Car
count = dataFrame['Car'].value_counts()
print"\nCount in column Car"
print(count)

# removing duplicates
dataFrame = dataFrame.drop_duplicates()
print"\nUpdated DataFrame after removing duplicates...\n",dataFrame

# counting frequency of column Car after removing duplicates
count = dataFrame['Car'].value_counts()
print"\nCount in column Car"
print(count)

Output

This will produce the following output −

Dataframe...
           Car        Place   UnitsSold
0          BMW        Delhi         95
1     Mercedes    Hyderabad         70
2  Lamborghini   Chandigarh         80
3          BMW        Delhi         95
4     Mercedes    Hyderabad         70
5      Porsche       Mumbai         90

Count in column Car
BMW            2
Mercedes       2
Porsche        1
Lamborghini    1
Name: Car, dtype: int64

Updated DataFrame after removing duplicates...
           Car         Place   UnitsSold
0          BMW        Delhi         95
1     Mercedes    Hyderabad         70
2  Lamborghini   Chandigarh         80
5      Porsche       Mumbai         90

Count in column Car
BMW           1
Porsche       1
Lamborghini   1
Mercedes      1
Name: Car, dtype: int64