PHP数组操作期间内存使用量突然增加

Question

I'm writing a maintenance script in PHP. The script keeps about 100,000 key-value pairs in an associative array and compare a bunch of other data with that array.

The keys are 12- or 16-byte hexademical strings.

The values are arrays containing 1-10 strings. Each string is around 50 bytes.

I'm populating my array by reading a text file line-by-line with fgets() in a loop.

All is fine until I hit about 44,000 keys, but after that the memory usage suddenly skyrockets.

No matter how much I increase the memory limit (and I'm relunctant to give it any more than 256MB at the moment), the memory usage increases exponentially until it hits the new limit. This is weird!

The following is a table with the number of keys on the left and the memory usage on the right.

10000     6668460
20000    12697828
30000    18917768
40000    25045068
41000    25658148
42000    26760304
43000    27350368
44000    27920400
45000    33438520
46000    77800344
47000   114203960
48000   161989660
49000   168419992
50000   206265572
Fatal error: Allowed memory size of 268435456 bytes exhausted

As you can see, the memory usage is consistent at 620-660 bytes per key until I reach 44,000 keys. After that, memory usage suddenly begins to increase, until it reaches over 4KB per key at 50,000 keys. This is very strange because the size of my keys and values are always similar.

It seems like I'm hitting some sort of internal limit on the number of keys I can have in an array, beyond which it all becomes very inefficient.

If I can maintain a memory usage of 620-600 bytes per key (which sounds reasonable given the usual overhead of using an array), my entire dataset should fit in approx. 64MB of memory and therefore easily accessible when I need to reference it later in the same script. This was the assumption when I first started writing the script. It's a maintenance script run from the CLI, so it's OK to use 64MB of memory from time to time.

But if the memory usage keeps increasing like the above, I'll have no choice but to offload the key-value dataset to an external daemon like Memcached, Redis, or an SQL database, and the network overhead will greatly slow down the maintenance script.

What I tried so far:

• I tried flattening the two-dimensional array into multiple one-dimensional array. No luck.
• I tried splitting the large array into multiple smaller arrays. No luck.
• I tried not using arrays at all and turning every key into a separate variable. No luck.
• I can't use SplFixedArray because my keys are not numeric (and can't be converted to numbers within the integer range) and the array needs to be mutable.
• I would rather not use Quickhash, Judy, or any of the other alternative array implementations written as a C extension.
• Sorry, this script needs to be in PHP. Don't ask me why...

The test server is a virtual machine running Ubuntu 12.04 LTS, 32-bit, with PHP 5.3.10-1ubuntu3.9.

Any ideas?

• Is this something that is fixed in a more recent version of PHP?
• Should I just give the dataset to an external daemon like Memcached?

Thanks!

Answer 1

It's the garbage collection I think. At some point you're using operations that allocate temporary space which then cannot be freed during the hard work so php will eat up your memory for no real purpose.

When I faced this problem I finally got to the conclusion that garbage is being thrown away at specific events only, like exiting a function. So what you should try is, make the job in several smaller steps and let your variables "relax" between them - make a function that only does a thousand elements at a time, then call it again to continue where it left off.

Hope this helps.