Eliminating Duplicates with uniq
If you’ve sorted files using the handy-dandy sort command, you might end up with results that have duplicates in them. Heck, you might have files with duplicates. At any rate, here’s how to find and work with them. As Code Listing 6.18 shows, you can get rid of duplicate lines by using the uniq command (short for “unique”) in conjunction with sort.
✓ Tips
uniq finds only identical, adjacent (sorted) lines. For example, if you have both Jones and jones in your address book, uniq won’t identify either entry because they differ in capitalization.
You can also use the -d flag to specify that you want to see only the duplicate lines. For example, say you want to see all of the people who are in both your carpool file ...
Get Unix Third Edition: Visual Quickstart Guide now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
