One of the repositories I maintain is a beginner’s GitHub repo. New developers can make their first pull request by adding their GitHub handle to a simple text file.
When pull requests get merged into the master branch, they often contain duplicates. The file has more than 7,000 lines. Names are not sorted alphabetically.
I needed a simple way to remove all duplicates lines from the file without sorting the lines.
I’m using awk, a Unix shell program. I’m not proficient in using awk, but I’ve found useful one-liners that do what I want.
For reference, this is how my file should look like:
# CONTRIBUTORS
- [@RupamG](https://github.com/RupamG)
- [@hariharen9](https://github.com/hariharen9)
- [@clevermiraz](https://github.com/clevermiraz)
- [@smeubank](https://github.com/smeubank)
- [@LJones95](https://github.com/LJones95)
- [@shannon-nz](https://github.com/shannon-nz)
- [@sammiepls](https://github.com/sammiepls)
Here’s how it often looks like:
# CONTRIBUTORS
- [@RupamG](https://github.com/RupamG)
- [@hariharen9](https://github.com/hariharen9)
- [@clevermiraz](https://github.com/clevermiraz)
- [@smeubank](https://github.com/smeubank)
- [@LJones95](https://github.com/LJones95)
- [@hariharen9](https://github.com/hariharen9)
- [@shannon-nz](https://github.com/shannon-nz)
- [@sammiepls](https://github.com/sammiepls)
- [@shannon-nz](https://github.com/shannon-nz)
1. Remove all empty lines
awk 'NF > 0' file.txt
NF is the Number of Fields Variable.
2. Remove duplicates
awk '!seen[$0]++' file.txt
I stole this command from opensource.com, where you can find an explanation on how it works.
3. Add Empty Lines Again
awk '{print; print "";}' file.txt
See Stackexchange.