C# program to remove all duplicates words from a given sentence

Removing duplicate words from a sentence is a common string manipulation task in C#. This process involves splitting the sentence into individual words, identifying duplicates, and keeping only unique words while preserving the original structure.

There are several approaches to accomplish this task, ranging from using LINQ's Distinct() method to using collections like HashSet for efficient duplicate removal.

Using LINQ Distinct() Method

The Distinct() method from LINQ provides a straightforward way to remove duplicates from a collection −

using System;
using System.Linq;

public class Program {
   public static void Main() {
      string str = "One Two Three One Four Two";
      string[] words = str.Split(' ');
      
      Console.WriteLine("Original sentence: " + str);
      
      // Remove duplicates and maintain original order
      var uniqueWords = words.Distinct().ToArray();
      
      string result = string.Join(" ", uniqueWords);
      Console.WriteLine("After removing duplicates: " + result);
   }
}

The output of the above code is −

Original sentence: One Two Three One Four Two
After removing duplicates: One Two Three Four

Using HashSet for Case-Insensitive Removal

When you need case-insensitive duplicate removal, using a HashSet with a custom comparer is more effective −

using System;
using System.Collections.Generic;
using System.Linq;

public class Program {
   public static void Main() {
      string str = "Hello world HELLO programming World";
      string[] words = str.Split(' ');
      
      Console.WriteLine("Original sentence: " + str);
      
      // Use HashSet with case-insensitive comparer
      HashSet<string> uniqueSet = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
      List<string> uniqueWords = new List<string>();
      
      foreach (string word in words) {
         if (uniqueSet.Add(word)) {
            uniqueWords.Add(word);
         }
      }
      
      string result = string.Join(" ", uniqueWords);
      Console.WriteLine("After removing duplicates: " + result);
   }
}

The output of the above code is −

Original sentence: Hello world HELLO programming World
After removing duplicates: Hello world programming

Using GroupBy for Advanced Processing

For more complex scenarios where you want to track duplicate counts or perform additional processing, GroupBy is useful −

using System;
using System.Linq;

public class Program {
   public static void Main() {
      string str = "apple banana apple orange banana grape";
      string[] words = str.Split(' ');
      
      Console.WriteLine("Original sentence: " + str);
      
      // Group by word and show counts
      var wordGroups = words.GroupBy(w => w.ToLower())
                           .Select(g => new { Word = g.First(), Count = g.Count() });
      
      Console.WriteLine("\nWord frequency:");
      foreach (var group in wordGroups) {
         Console.WriteLine($"{group.Word}: {group.Count}");
      }
      
      // Get unique words only
      string result = string.Join(" ", wordGroups.Select(g => g.Word));
      Console.WriteLine("\nAfter removing duplicates: " + result);
   }
}

The output of the above code is −

Original sentence: apple banana apple orange banana grape

Word frequency:
apple: 2
banana: 2
orange: 1
grape: 1

After removing duplicates: apple banana orange grape

Comparison of Methods

Conclusion

Removing duplicate words from a sentence in C# can be accomplished using LINQ's Distinct() method, HashSet collections, or GroupBy operations. Choose the method based on your specific requirements for case sensitivity, performance, and order preservation.