使用php在mysql中插入一百万行时如何防止内存不足

Question

I have built a script in Laravel that reads a JSON file line by line and imports the contents into my database.

However, when running the script, I get an out of memory error after inserting about 80K records.

mmap() failed: [12] Cannot allocate memory

mmap() failed: [12] Cannot allocate memory
PHP Fatal error:  Out of memory (allocated 421527552) (tried to allocate 12288 bytes) in /home/vagrant/Code/sandbox/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php on line 1758

mmap() failed: [12] Cannot allocate memory
PHP Fatal error:  Out of memory (allocated 421527552) (tried to allocate 32768 bytes) in /home/vagrant/Code/sandbox/vendor/symfony/debug/Exception/FatalErrorException.php on line 1

I have built a sort of makeshift queue to only commit the collected items every 100, but this made no difference.

This is what the part of my code that does the inserts looks like:

public function callback($json) {

    if($json) {

        $this->queue[] = [

            'type' => serialize($json['type']),
            'properties' => serialize($json['properties']),
            'geometry' => serialize($json['geometry'])
        ];

        if ( count($this->queue) == $this->queueLength ) {

            DB::table('features')->insert( $this->queue );

            $this->queue = [];
        }
    }
}

It's the actual inserts ( DB::table('features')->insert( $this->queue ); ) that are causing the error, if I leave those out I can perfectly iterate over all lines and echo them out without any performance issues.

I guess I could allocate more memory, but doubt this would be a solution because I'm trying to insert 3 million records and it's currently already failing after 80K with 512Mb memory allocated. Furthermore, I actually want to run this script on a low budget server.

The time it takes for this script to run is not of any concern, so if I could somehow slow the insertion of records down that would be a solution I could settle for.

Answer 1

If you use MySQL 5.7+, it has feature for importing data from JSON file using LOAD DATA INFILE .

LOAD DATA INFILE 'afile.json' INTO TABLE atable (field1, field2, ....);

If you use lower MySQL version then you need to convert JSON into CSV format first. For example using https://github.com/danmandle/JSON2CSV

LOAD DATA INFILE 'afile.csv' INTO TABLE atable (field1, field2, ....);

See LOAD DATA INFILE 's documentation .

Answer 2

Check to make sure your query logging is off. If your query logging is on, even though you are clearing out $this->queue, the query log is growing until it grows to an unmanageable size and you hit your memory limit.

To check if query logging is on or off, use the DB facade like this:

DB::logging()

If this returns true, then your logging is on, and that's your problem. If this returns false, then it's some other configuration that is holding onto your queries.

Answer 3

About problem, I think you should use message queue ( Gearman, Rabbit ... ) to fix this problem.

Fetch all record and push to queue. Queue will process one by one

Answer 4

On scaffolding my application I installed the itsgoingd/clockwork package. It's a habit I've gotten into because it's such a useful tool. (I recommend any Laravel devs reading this to check it out!)

Of course this package logs queries for debugging purposes and thus, was the culprit eating up all of my memory.

Disabling the package by deleting the reference to its service provider in my configuration solved the issue.

Cheers to @dbushy727 for giving me a push in the right direction to figure this out.