How to Remove Duplicate Lines from a List (3 Methods)

August 22, 2026 · 4 min read

Duplicate entries in lists are a common headache. Whether you're cleaning up an email list, deduplicating keywords for SEO, or processing data exports, you need a fast way to keep only unique entries. Here are three methods, from easiest to most powerful.

METHOD 1 — FASTEST

Use a Free Online Tool

The fastest way is to paste your list into an online deduplication tool. No formulas, no commands — just paste, click, and copy the result.

Our Remove Duplicate Lines tool handles thousands of lines instantly, with options for case sensitivity, whitespace trimming, and alphabetical sorting. It runs in your browser so your data stays private.

Remove Duplicates Online →

METHOD 2 — SPREADSHEETS

Remove Duplicates in Excel or Google Sheets

In Excel: Select your data column, go to the Data tab, and click "Remove Duplicates." Excel tells you how many duplicates were found and removed. This modifies your data in place, so make a copy first.

In Google Sheets: Select your range, go to Data → Data Cleanup → Remove Duplicates. Alternatively, use the formula =UNIQUE(A1:A100) in a new column to extract unique values without modifying the original data.

METHOD 3 — COMMAND LINE

Using sort and uniq (Mac/Linux Terminal)

For large files or automated workflows, the command line is the most powerful option:

sort input.txt | uniq > output.txt

This sorts the file alphabetically and removes consecutive duplicate lines. To remove duplicates without sorting (preserving original order):

awk '!seen[$0]++' input.txt > output.txt

The awk command keeps the first occurrence of each line and removes all subsequent duplicates, without changing the order.

Which Method Should You Use?

For quick, one-off tasks with lists under 10,000 lines, the online tool is the fastest option — no setup needed. For structured data in spreadsheets, use Excel or Google Sheets built-in features. For large files (100K+ lines) or automated pipelines, the command line is the way to go.

Need to also count the words in your cleaned-up list? Try our Word Counter. Working with regex patterns to find and clean data? Check out the Regex Tester.