How to remove duplicates from MongoDB Collection?

To remove duplicates from a MongoDB collection, create a unique index on the field that should not have duplicate values. This prevents duplicate insertion and removes existing duplicates when combined with the dropDups option.

Syntax

db.collectionName.createIndex(
    { fieldName: 1 }, 
    { unique: true, dropDups: true }
);

Example: Preventing Duplicate FirstNames

Let's create a unique index on the FirstName field to prevent duplicates ?

db.demo604.createIndex(
    { FirstName: 1 }, 
    { unique: true, dropDups: true }
);

{
    "createdCollectionAutomatically" : true,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}

Testing Duplicate Prevention

Insert documents with unique FirstNames ?

db.demo604.insertMany([
    { FirstName: "Chris" },
    { FirstName: "Bob" },
    { FirstName: "David" }
]);

{
    "acknowledged" : true,
    "insertedIds" : [
        ObjectId("5e960887ed011c280a0905d8"),
        ObjectId("5e96088aed011c280a0905d9"),
        ObjectId("5e96088ded011c280a0905da")
    ]
}

Now try inserting a duplicate FirstName ?

db.demo604.insertOne({ FirstName: "Chris" });

WriteError: E11000 duplicate key error collection: test.demo604 index: FirstName_1 dup key: { : "Chris" }

Verify Results

Display all documents to confirm no duplicates exist ?

db.demo604.find();

{ "_id" : ObjectId("5e960887ed011c280a0905d8"), "FirstName" : "Chris" }
{ "_id" : ObjectId("5e96088aed011c280a0905d9"), "FirstName" : "Bob" }
{ "_id" : ObjectId("5e96088ded011c280a0905da"), "FirstName" : "David" }

Key Points

• unique: true prevents future duplicate insertions
• dropDups: true removes existing duplicates during index creation
• Use createIndex() instead of the deprecated ensureIndex()

Conclusion

Creating a unique index with dropDups: true effectively removes existing duplicates and prevents future duplicate insertions. The database automatically enforces uniqueness constraints through the index.