MySQL - How to prevent duplicates using LOAD DATA INFILE

Question

I believe I have an issue where a "LOAD DATA INFILE...REPLACE INTO TABLE" process is removing/replacing the duplicate information, but I think it's replacing the data by adding a new record with an updated ID.

In other words, if I have a table that looks like this:

ID | INFO | DATE       |
1  | foo | 2011-12-01  |
2  | bar | 2011-12-01  |

and I use LOAD DATA INFILE ...REPLACE that attempts to insert the INFO and DATE fields several more times as part of a process that inserts into other tables, I end up with something like below:

ID | INFO | DATE       |
15  | foo | 2011-12-01  |
23  | bar | 2011-12-01  |

So what happened is that it replaced the original data with one of the new duplicates and then kept the record with the new ID (the latest ID, I think??!!).

Here the code I'm using to manage this:

$sqlDomain = "LOAD DATA LOCAL INFILE '".$csvfile."' REPLACE INTO TABLE sg_domains FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (Domain,@dummy1,@dummy2,@dummy3,@dummy4,@dummy5) SET UserID = ".$UserID;

$sqlDelDupeDoms = "delete t2 from sg_domains as t1, sg_domains as t2 where t1.Domain = t2.Domain and t2.id > t1.id;";

$sqlDelLikeDoms = "delete t2 from sg_domains as t1, sg_domains as t2 where t1.Domain LIKE CONCAT('%',t2.Domain,'%') and t2.id > t1.id;";

The @dummy variables are intended to divert unwanted fields from attempting to be inserted into the table requiring only one field from the CSV file being used for other INSERT process later on. I'm also obviously using the two latter commands to "clean" out dupes and like/similar data on the fly.

Am I doing this correctly and/or efficiently???

Any advice would be appreciated.

Thanks!!!!!

Answer 1

That's how replace works. Maybe you want insert ignore or insert ... on duplicate key update ?

http://dev.mysql.com/doc/refman/5.5/en/replace.html

http://dev.mysql.com/doc/refman/5.5/en/insert.html