How to removes duplicate lines from a sorted file in Linux?

LinuxOpen SourceOperating System

To remove duplicate lines from a sorted file and make it unique, we use the uniq command in the Linux system. The uniq command work as a kind of filter program that reports out the duplicate lines in a file. It filters adjacent matching lines from the input and gives a unique output. This command is also available in the Windows and IBM i operating system.

Syntax

The general syntax of the uniq command is as follows

uniq [OPTION]... [INPUT [OUTPUT]]

Brief description of options available in the fmt command.

Sr.No.Option & Description
1-c, --count
Display how many times line was repeated.
2-d—repeated
Display only repeated lines, one for each group.
3-D
Display all duplicate lines.
4-f, --skip-fields=N
Avoid comparing the first N fields.
5-i, --ignore-case
While comparing ignore differences in case.
6-s, --skip-chars=N
Avoid comparing the first N characters.
7-u, --unique
Prints only unique lines
8-w, --check-chars=N
Line delimiter is NULL, not newline
9-v, --verbose
Compare no more than N characters in lines.
10--help
Display help and exit.
11--version
Output the version information and exit.

To print after removal repeated lines in a file, we use the uniq command in the Linux system as shown below.

$ cat >text.txt
Print only unique lines.
The earth is round.
The earth is round.
Welcome to the tutorialpoint...
Welcome to the tutorialspint...
$ uniq text.txt
Print only unique lines.
The earth is round.
Welcome to the tutorialpoint...

To print number of duplicate lines of a file, we use the -c or --count option with the unique command as shown below.

$ uniq –c text.txt
2 The earth is round.
2 Welcome to the tutorialspoint...
1 Print only unique lines.

To prints only unique lines of a file, we use the -u or –unique option with the uniq command as shown below.

$ uniq –u text.txt
Print only unique lines.

To check more information about the uniq command, we use the --help option with the uniq command in the Linux operating system as shown below.

$ uniq --help

To check version information of the uniq command, we use the --version option with the uniq command in the Linux operating system as shown below.

$ uniq --version
raja
Published on 01-Jul-2021 15:33:37
Advertisements