Remove Duplicate Line in Vim?

Question

I\'m trying to use VIM to remove a duplicate line in an XML file I created. (I can\'t recreate the file because the ID numbers will change.)

The file looks something

Answer 1

You could select the lines then do a :'<,'>sort u if you don't care about the ordering. It will sort and remove duplicates.

Answer 2

to the OP, if you have bash 4.0

#!/bin/bash
# use associative array
declare -A DUP
file="myfile.txt"
while read -r line
do
    if [ -z ${DUP[$line]} ];then
        DUP[$line]=1
        echo $line >temp
    fi
done < "$file"
mv temp "$file"

Answer 3

with python to remove all repeated lines:

#!/usr/bin/env python

import sys
def remove_identical(filein, fileout) : 
  lines = list()
  for line in open(filein, 'r').readlines() : 
    if line not in lines : lines.append(line)
  fout = open(fileout, 'w')
  fout.write(''.join(lines))
  fout.close()

remove_identical(sys.argv[1], sys.argv[2])