CodeIgniter DataMapper ORM memory issue

Question

I am using CodeIgniter and DataMapper to manage relationships for an ecommerce store. The relationships are:

A product has many skus A product has many amazon_products A sku has many amazon_products

I'm trying to retrieve all of the data from the database and format it to be used by Backbone on the front end. I ultimately want to send json to the page in the following format:

Array of products
    Product
        Array of amazon_products 
        Array of skus
            Array of amazon_products
    Product
        Array of amazon_products 
        Array of skus
            Array of amazon_products
    Product
        Array of amazon_products 
        Array of skus
            Array of amazon_products

Currently, I'm generating this data with the following PHP code:

$products = new Product();
$products->order_by('ext_created_on', 'desc')->get();

$data['products'] = array();

foreach($products as $product){
        $product_array = $product->to_array();
        $product_array['amazon'] = $product->amazon_product->get()->all_to_array();
        foreach($product->sku->get() as $sku){
                $sku_array = $sku->to_array();
                $sku_array['amazon'] = $sku->amazon_product->get()->all_to_array();
                $product_array['skus'][] = $sku_array;
        }
        $data['products'][] = $product_array;

        echo memory_get_usage()."<br>"
}

$data['products'] = json_encode($data['products']);

When I monitor the memory of my Ubuntu server when this executes, it seems to be using a lot (~13% of Memory). Also, since there are about 700 products in the database, DataMapper is running 1000's of queries to get all of this data. Are the queries causing the high memory use? Or is there something I'm overlooking in the foreach loop that could be resulting in a memory a memory leak?

The output of get_memory_usage() is 33678176 after the first product loop, and 109580968 after the last.

Answer 1

In case anyone comes across this, I was able to cut the memory usage to about 1/3rd by using DataMapper's clear() method. I simply added the line $product->clear() at the end of the outer foreach loop.

I also added a $sku->clear() at the end of the inner loop.

Answer 2

While I've never used the DataMapper library you are using, you mentioned that there are thousands of queries being run in order to retrieve the results. Based on that, it seems like maybe there is a better method than get() for retrieving $product->sku and $sku->amazon_product. Is your first get() on line #2 not acquiring all of the information for you? My thinking is that get() does not check for available result values, and that another function in your DataMapper may refer to the available result set before making additional calls to the database.

In fact, I wonder if $product->sku is already cached with the value for sku.