PHP Detect Duplicate Text

Question

I have a site where users can put in a description about themselves.

Most users write something appropriate but some just copy/paste the same text a number of times (to

Answer 1

This is a basic text classification problem. There are lots of articles out there on how to determine if some text is spam/not spam which I'd recommend digging into if you really want to get into the details. A lot of it is probably overkill for what you need to do here.

Granted one approach would be to evaluate why you're requiring people to enter longer bios, but I'll assume you've already decided that forcing people to enter more text is the way to go.

Here's an outline of what I would do:

1. Build a histogram of word occurrences for the input string
2. Study the histograms of some valid and invalid text
3. Come up with a formula for classifying a histogram as valid or not

This approach would require you to figure out what's different between the two sets. Intuitively, I'd expect spam to show fewer unique words and if you plot the histogram values, a higher area under the curve concentrated toward the top words.

Here's some sample code to get you going:

$str = 'Love a and peace love a and peace love a and peace love a and peace love a and peace love a and peace';

// Build a histogram mapping words to occurrence counts
$hist = array();

// Split on any number of consecutive whitespace characters
foreach (preg_split('/\s+/', $str) as $word)
{
  // Force all words lowercase to ignore capitalization differences
  $word = strtolower($word);

  // Count occurrences of the word
  if (isset($hist[$word]))
  {
    $hist[$word]++;
  }
  else
  {
    $hist[$word] = 1;
  }
}

// Once you're done, extract only the counts
$vals = array_values($hist);
rsort($vals); // Sort max to min

// Now that you have the counts, analyze and decide valid/invalid
var_dump($vals);

When you run this code on some repetitive strings, you'll see the difference. Here's a plot of the $vals array from the example string you gave:

Compare that with the first two paragraphs of Martin Luther King Jr.'s bio from Wikipedia:

A long tail indicates lots of unique words. There's still some repetition, but the general shape shows some variation.

FYI, PHP has a stats package you can install if you're going to be doing lots of math like standard deviation, distribution modeling, etc.