Updating 11M rows database and my.cnf optimisation

Question

I have to update 11M rows from a database in a PHP script.

After some time, the script freezes or crashes. I have to restart EasyPHP 12, and reload it.

My configuration:

• Windows 7 Pro 64 bits
• Intel Core i7 860 2.8Ghz
• 8G RAM

My my.cnf file :

port        = 3306
socket      = /tmp/mysql.sock

[mysqld]
port        = 3306
socket      = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 384M
max_allowed_packet = 1M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
thread_concurrency = 8
log-bin=mysql-bin
server-id = 1

[mysqldump]
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

Here is the pseudo code that crash.

`For i to 100000 { do magic (check content on the web);    UPDATE table;    }`

Answer 1

You have to look into php.ini, not my.cnf.

I suppose you are performing some logic on a single record and updating that record, then on to the next one. In this case the update of a single record (or a subset of records) should not take that long.

The freeze or crash is either because your script hits the memory limit or its execution time limit.

Answer 2

You are probably running into an execution time error. You could create a shell script an run that via your command line.

for example (pseudo):

<?php
SELECT * FROM table
LIMIT $x to $y

FOR every result
do some magic, SAVE to record with ID $id

If you call this file my_update.php , run it by typing php path/to/my_update.php and watch the magic. (where php is the php executable).

Be smart and log every action! So when the script fails you have a nice trail and don't have to start all over again! That is exactly why I added the LIMIT in the query so it doesn't have to buffer 11M of rows, but just a few. After the first bunch of rows it will simply go to the next LIMIT. Sort of like pagination, but without visual output.

Sources:

• http://php.net/manual/en/features.commandline.php

Answer 3

您需要从php.ini文件中更改脚本的最大执行时间。