Using php/MySQL to run queries on 3+million rows

Question

I have been tasked with creating a web page which queries a table which has 3 million rows. I don't want to print all of the rows, but instead to create a report which lists the top 10 items which appear in the same basket as another item.

I haven't ever used php to deal with this amount of data before so my question is not regarding the actual coding to make the calculations, but what are the key considerations I should take with regards to Should php or sql do the bulk of the calculation, The page will not be updating any records but only making calculations based on their values and frequency How expensive is this in terms of timeout concerns, hardware. Etc

The page won't be loaded at a huge volume, I'd imagine less than 1000 a day.

Answer 1

I think what you need to pay attention to are

MySQL: proper index created for column you need to do where or join , use explain to see if query are using the right index . You may also want to look at partitions table but for 3 millon I do not think it is necessary unless your data is keep growing fast.

PHP: make sure you are not loading too much data to memory , you can use xdebug to dump memory usage file to check if you have slow down problem.