comparing the contents of two huge text files quickly

Question

what i\'m basically trying to do is compare two HUGE text files and if they match write out a string, i have this written but it\'s extremely slow. I was hoping you guys might

Answer 1

First I'd suggest normalizing both files and putting one of them in a set. This allows you to quickly test whether a specific line is present and reduces the complexity from O(n*n) to O(n).

Also you shouldn't open and close the file every time you write a line:

File.AppendAllText(...); // This causes the file to be opened and closed.

Open the output file once at the start of the operation, write lines to it, then close it when all lines have been written.

Answer 2

• Call File.ReadLines() (.NET 4) instead of ReadAllLines() (.NET 2.0).
  ReadAllLines needs to build an array to hold the return value, which can be extremely slow for large files.
  If you're not using .Net 4.0, replace it with a StreamReader.

• Build a Dictionary<string, string> with the matchCollects (once), then loop through the foundList and check whether the HashSet contains matchFound.
  This allows you to replace the O(n) inner loop with an O(1) hash check

• Use a StreamWriter instead of calling AppendText

• EDIT: Call Path.GetFileNameWithoutExtension and the other Path methods instead of manually manipulating strings.

For example:

var collection = File.ReadLines(@"C:\found.txt")
    .ToDictionary(s => s.Split('\\')[3].Replace(".txt", ""));

using (var writer = new StreamWriter(@"C:\Copy.txt")) {
    foreach (string found in foundlist) {
        string splitFound = found.Split('|');
        string matchFound = Path.GetFileNameWithoutExtension(found)

        string collectedLine;
        if (collection.TryGetValue(matchFound, collectedLine)) {
            end++;
            long finaldest = (start - end);
            Console.WriteLine(finaldest);
            writer.WriteLine("copy \"" + collectedLine + "\" \"C:\\OUT\\" 
                           + splitFound[1] + "\\" + spltifound[0] + ".txt\"");
        }
    }
}

Answer 3

You have a cartesian product, so it makes sense to index one side instead of doing an enhaustive linear search.

Extract the keys from one file and use either a Set or SortedList data structure to hold them. This will make the lookups much much faster. (Your overall algorithm will be O(N lg N) instead of O(N**2) )