How to insert a million records into a mysql database?

Question

I have a two tables.

TABLE 1 has id(PRIMARY)|RANK|WEBSITE |ADDRESS
This is filled with a million websites.

TABLE 2 has id(PRIMARY)|tag1|tag2|tag3 ......|tag30

I am generating the tags by scraping for the meta tags and other attributes.

I have no problems populating the database for the first few thousand websites. After that, I am not able to populate it. I guess it timed out.

How should I proceed to populate the entire database?

EDIT:

I am getting the data by scraping the urls of the websites dynamically

Answer 1

It would be good if you specify error message in your question, output of mysql_error etc (if any).

If your script execution is interrupted because of PHP timeout, you may try to set corresponding configuration parameters (see here for php.ini configuration for details, and here for configuration in runtime).

If reconfiguration is not possible, you may store last record ID you updated in file / database table, and continue execution starting from next ID next time you run your PHP script.