How can I remove duplicate lines in Visual Studio Code?

Question

Say you have the following text:

abc
123
abc
456
789
abc
abc

I want to remove all \"abc\" lines and just keep one. I don\'t mind sorting. T

Answer 1

To add to @Marc.2377 's reply.

If the order is important and you don't care that you just keep the last of the duplicate lines, simply search for the following regexp if you want to only remove duplicte non-empty lines

^(.+\n)(?=(?:.*\n)*?\1)

If you also want to remove duplicate empty lines, use * instead of +

^(.*\n)(?=(?:.*\n)*?\1)

and replace with nothing.

This will take a line and try to find ahead some more (maybe 0) lines followed by the exact same line taken. It will remove the taken line.

This is just a one-shot regex. No need to spam the replace button.

Answer 2

Not actually in Visual Studio Code, but if it works, it works.

1. Open a new Excel spreadsheet
2. Paste the data into a column
3. Go to the Data tab
4. Select the column of data (if you haven't already)
5. Click Remove Duplicates (somewhat in the middle of the bar)
6. Click OK to remove duplicates.

It is not the best answer, as you specified Visual Studio Code, but as I said: If it works, it works :)

Answer 3

Sublime Text 3

It has blisteringly fast native permutation functions.

• Edit > Permute Lines > Unique or ⇧⌘U, and
• Edit > Permute Selections > Unique

Visual Studio Code is my daily driver. But, I keep Sublime Text on standby for these situations.

Answer 4

Try find and replace with a regular expression.

• Find: ^(.+)((?:\r?\n.*)*)(?:\r?\n\1)$

• Replace: $1$2

It is possible to introduce some variance in the first group.

Answer 5

Here is a very interesting extension: Transformer

Features:

• Unique Lines
• Unique Lines As New Document
• Filter Lines
• Filter Lines As New Document
• Sort Lines
• Sort Lines By Length
• Align To Cursor
• Align CSV
• Compact CSV
• Copy To New Document
• Select Lines
• Lines As JSON
• Trim Lines
• Count Duplicate Lines As New Document
• Macros

For removing duplicate lines:

• Removes duplicate lines from the document

• Operates on selection or current block if no selection

I haven't played with it much besides the "Unique Lines" command but it seems quite nicely done (including attempting a macro recorder!).

Answer 6

I just had the same issue and found the Visual Studio Code package "Sort lines". See the Visual Studio Code market place for details (e.g. Sort lines).

This package has the option "Sorting lines (unique)", which did it for me. Take care of any white spaces at the beginning/end of lines. They influence whether lines are considered unique or not.