Text comparison algorithm

Question

We have a requirement in the project that we have to compare two texts (update1, update2) and come up with an algorithm to define how many words and how many sentences have

Answer 1

Some kind of diff variant might be helpful, eg wdiff

If you decide to devise your own algorithm, you're going to have to address the situation where a sentence has been inserted. For example for the following two documents:

The men are bad. I hate the men

and

The men are bad. John likes the men. I hate the men

Your tool should be able to look ahead to recognise that in the second, I hate the men has not been replaced by John likes the men but instead is untouched, and a new sentence inserted before it. i.e. it should report the insertion of a sentence, not the changing of four words followed by a new sentence.

Answer 2

The specific algorithm used by diff and most other comparison utilities is Eugene Myer's An O(ND) Difference Algorithm and Its Variations. There's a Java implementation of it available in the java-diff-utils package.

Answer 3

The difficulty comes when comparing large files efficiently and with good performance. I therefore implemented a variation of Myers O(ND) diff algorithm - which performs quite well and accurate (and supports filtering based on regular expression):

Algorithm can be tested out here: becke.ch compare tool web application

And a little bit more information on the home page: becke.ch compare tool

Answer 4

Typically this is accomplished by finding the Longest Common Subsequence (commonly called the LCS problem). This is how tools like diff work. Of course, diff is a line-oriented tool, and it sounds like your needs are somewhat different. However, I'm assuming that you've already constructed some way to compare words and sentences.

Answer 5

An O(NP) Sequence Comparison Algorithm is used by subversion's diff engine.

For your information, there are implementations with various programming languages by myself in following page of github.

https://github.com/cubicdaiya/onp

Answer 6

Here are two papers that describe other text comparison algorithms that should generally output 'better' (e.g. smaller, more meaningful) differences:

• Tichy, Walter F., "The String-to-String Correction Problem with Block Moves" (1983). Computer Science Technical Reports. Paper 378.
• Paul Heckel, "A Technique for Isolating Differences Between Files", Communications of the ACM, April 1978, Volume 21, Number 4

The first paper cites the second and mentions this about its algorithm:

Heckel[3] pointed out similar problems with LCS techniques and proposed a linear-lime algorithm to detect block moves. The algorithm performs adequately if there are few duplicate symbols in the strings. However, the algorithm gives poor results otherwise. For example, given the two strings aabb and bbaa, Heckel's algorithm fails to discover any common substring.

The first paper was mentioned in this answer and the second in this answer, both to the similar SO question:

• Is there a diff-like algorithm that handles moving block of lines? - Stack Overflow