If you have a file with repeated lines, and you want to remove duplicate lines, you can use the uniq command. The description of this utility, from its man page is simply: “report or omit repeated lines.” Consider the following example:
trevor@MyUbuntuVM:~$ echo "this is is great" | uniq
The output of the above command looks like:
this is great
You can also get a count of duplicate lines by adding on the -c parameter to uniq. The following command …
trevor@MyUbuntuVM:~$ echo "this is is great" | uniq -c
… emits this output.
1 this 2 is 1 great