Best way to avoid duplicate entry into mysql database

Question

I have a table with 3 columns - id (pk), pageId (fk), name. I have a php script which dumps about 5000 records into the table, with about half being duplicates, with same pa

Answer 1

From a mysql point you can do

alter table YOURTABLE add unique index(pageId, name);

If your wording is correct and you want to do it from php you can do

$already_done = array();
foreach ($records as $record)
{
   $unique_hash = md5($record['name'].$record['pageId']);
   if (!in_array($unique_hash, $already_done))
   {
      $already_done[] = $unique_hash;
      // sql insert here
   }
}

either way those should do you just fine.

Answer 2

First step would be to set a unique key on the table:

ALTER TABLE thetable ADD UNIQUE INDEX(pageid, name);

Then you have to decide what you want to do when there's a duplicate. Should you:

1. ignore it?

   INSERT IGNORE INTO thetable (pageid, name) VALUES (1, "foo"), (1, "foo");
   

2. Overwrite the previously entered record?

   INSERT INTO thetable (pageid, name, somefield)
   VALUES (1, "foo", "first")
   ON DUPLICATE KEY UPDATE (somefield = 'first')
   
   INSERT INTO thetable (pageid, name, somefield)
   VALUES (1, "foo", "second")
   ON DUPLICATE KEY UPDATE (somefield = 'second')
   

3. Update some counter?

   INSERT INTO thetable (pageid, name)
   VALUES (1, "foo"), (1, "foo")
   ON DUPLICATE KEY UPDATE (pagecount = pagecount + 1)
   

Answer 3

You can also ignore the error with mysql: INSERT IGNORE INTO TABLE ... it will ignore the key error, skip over that insert and move on to the next.

Answer 4

You can set the PageID and Name to a Unique index in the MySQL database. This way when you insert the rows, it will cause an error, which can be ignored by PHP, and you can just go to the next row.

This assumes you are inserting rows individually. AKA:

foreach($large_data as $fields)
{
    mysql_query("INSERT INTO TABLE (`Something`) VALUES('".$fields['something']."');
}