mySQL：如何基于四个字段识别重复项

Question

I have read a few posts on SO on how to delete duplicates, by comparing a table with another instance of itself, however I don't want to delete the duplicates I want to compare them.

eg. I have the fields "id", "sold_price", "bruksareal", "kommunenr", "Gårdsnr" ,"Bruksnr", "Festenr", "Seksjonsnr". All fields are int.

I want to identify the rows that are duplicates/identical (the same bruksareal, kommunenr, gårdsnr, bruksnr,festenr and seksjonsnr). If identical then I want to give these rows a unique reference number.

I believe this will make is easier to identify the rows that I later want to compare on other fields (eg. such as "sold_price", "sold_date" etc..)

I'm open to suggestions if you believe my approach is wrong...

Answer 1

Perform a join on the table to itself across all fields, then use an exists , query, such as:

Update Table1
Set reference = UUID()
Where exists (
 Select tb1.id
 from Table1 tb1 inner join Table1 tb2 on
  tb1.Field1 = tb2.Field1 AND
  tb1.Field2 = tb2.Field2 AND
  etc
 Where tb1.Id = Table1.Id
 And tb1.Id != tb2.Id
)

actually you can simplify with just a join

Update Table1
Set reference = UUID()
From Table1 inner join Table1 tb2 on
      Table1.Field1 = tb2.Field1 AND
      Table1.Field2 = tb2.Field2 AND
      etc
Where Table1.Id != tb2.Id

Answer 2

Depending on where you want to do that, i would go for a hash implementation. For every insert, calculate the hash of the needed columns when you do the insert (trigger maybe), and after that you should be able to find out very easily what rows are duplicated (if you index that column, the queries should be pretty fast, but remember that that is still not a int column, so it will get a little slower over time).

After this you can do whatever you please with the duplicated records, without very expensive queries on the database.

Later edit: Make sure that you convert the null values into some defined value, since some of the mysql functions like MD5 will just return null if the operand is null. The same goes for concat - if one operand is null, it will return null (the same is not valid for concat_ws though).