Remove duplicates in large MySql table

Question

I have a question about MySql. I have a table with 7.479.194 records. Some records are duplicated. I would like to do this:

insert into new_table 
  select *         


        

Answer 1

To avoid the memory issue, avoid the big select by having a small external program, using the logic as below. First, backup your database. Then:

do {
# find a record
x=sql: select * from table1 limit 1;
if (null x)
then
 exit # no more data in table1
fi
insert x into table2

# find the value of the field that should NOT be duplicated
a=parse(x for table1.a)
# delete all such entries from table1
sql: delete * from table1 where a='$a';

}