perl script to create xml from mysql query - out of memory

Question

I need to generate an XML file from database records, and I get the error "out of memory". Here's the script I am using, it's found on Google, but it's not suitable for me, and it's also killing the server's allocated memory. It's a start though.

#!/usr/bin/perl

use warnings;
use strict;
use XML::Simple;
use DBI;

my $dbh = DBI->connect('DBI:mysql:db_name;host=host_address','db_user','db_pass')
  or die DBI->errstr;

# Get an array of hashes
my $recs = $dbh->selectall_arrayref('SELECT * FROM my_table',{ Columns => {} });

# Convert to XML where each hash element becomes an XML element
my $xml = XMLout( {record => $recs}, NoAttr => 1 );

print $xml;

$dbh->disconnect;

This script only prints the records, because I tested with a where clause for a single row id.

• First of all, I couldn't manage to make it to save the output to a file.xml.

• Second, I need somehow to split the "job" in multiple jobs and then put together the XML file all in one piece.

I have no idea how to achieve both.

Constraint: No access to change server settings.

Answer 1

These are problem lines:

my $recs = $dbh->selectall_arrayref('SELECT * FROM my_table',{ Columns => {} });

This reads the whole table into memory, representing every single row as an array of values.

my $xml = XMLout( {record => $recs}, NoAttr => 1 );

This is probably even larger structure, it is a the whole XML string in one go.

The lowest memory-use solution needs to involve loading the table one item at a time, and printing that item out immediately. In DBI, it is possible to make a query so that you fetch one row at a time in a loop.

You will need to play with this before the result looks like your intended output (I haven't tried to match your XML::Simple output - I'm leaving that to you:

print "<records>\n";

my $sth = $dbh->prepare('SELECT * FROM my_table');
$sth->execute;

while ( my $row = $sth->fetchrow_arrayref ) {
  # Convert db row to XML row
  print XMLout( {row => $row}, NoAttr => 1 ),"\n";
}

print "</records>\n";

Perl can use eg open( FILEHANDLE, mode, filename ) to start access to a file and print FILEHANDLE $string to print to it, or you could just call your script and pipe it to a file eg perl myscript.pl > table.xml

Answer 2

It's the select * with no contraints that will be killing your memory. Add some constraint to your query ie date or id and use a loop to execute the query and do your output in chunks. That way you won't need to load the whole table in mem before your get started on the output.