Remove duplicate lines from a list of lines – on a Mac

Today I had a list consisting of 16,000 lines where most where duplicates. The list was in Google Spreadsheets (I’m allergic to Excel on a Mac) and I wanted to remove all the duplicates from the list and only get a much shorter list of the unique values. Now this is amazingly hard to do in Google Docs, and none of my local text editors had a feature for it either.

After complaing to my tech colleagues in Berlin, there turned out to be an incredibly fast way to do it on a Mac.

This is how to do it:

1. Just save the list in a .txt file on your local drive. I saved it as UnsortedLongList.txt
2. Open the Terminal.
3. Go to the directory where you saved the file. In my case in Documents, so cd Documents took me there.
4. Type the following command into Terminal: sort UnsortedLongList.txt| uniq > NoDupes.txt
5. That’s it!

Now you have a txt document with a much shorter list consisting of the unique values only.

Thanks Jon! You saved me a lot of time in the future. I have to remember to ask stuff more often.