Large MySQL Table daily updates?

Question

I have a MySQL table that has a bunch of product pricing information on around 2 million products. Every day I have to update this information for any products whose pricing information has changed [huge pain].

I am wondering what the best way to handle these changes are other than running something like compare and update any products that have changed ?

Love any advice that you can provide

Answer 1

For bulk updates you should definitely be using LOAD DATA INFILE rather than a lot of smaller update statements.

First, load the new data into a temporary table:

LOAD DATA INFILE 'foo.txt' INTO TABLE bar (productid, info);

Then run the update:

UPDATE products, bar SET products.info = bar.info WHERE products.productid = bar.productid;

If you also want to INSERT new records from the same file that you're updating from, you can SELECT INTO OUTFILE all of the records that don't have a matching ID in the existing table then load that outfile into your products table using LOAD DATA INFILE .

Answer 2

I maintain a price comparison engine with millions of prices and I select each row that I find in the source and update each row individually. If there is no row then I insert. Its best to use InnoDB transactions to speed this up.

This is all done by a PHP script that knows how to parse the source files and update the tables.