使用Codeigniter优化CSV导出

Question

i have recently migrated my export CSV code from core php to code igniter. the code works well but its very slow when exporting very large amount of data..

here is my old code:

function exportCSV($qry,$con,$title)
{
$result = mysql_query($qry, $con) or die(mysql_error($con));

header('Content-Type: text/csv; charset=UTF-8');
header("Cache-Control: no-store, no-cache");  
header("Content-Disposition: attachment;filename=".$title."-".date('mdY').".csv");
//echo "\xEF\xBB\xBF";

$row = mysql_fetch_assoc($result);
 if ($row) {
  echocsv(array_keys($row));
 }
 while ($row) {
  echocsv($row);
  $row = mysql_fetch_assoc($result);
 }
}

function echocsv($fields)
{
    $separator = '';
    foreach ($fields as $field) {
        if (preg_match('/\\r|\\n|,|"/', $field)) {
            $field = '"' . str_replace('"', '""', $field) . '"';
        }
        echo $separator . $field;
        $separator = ',';
    }
    echo "\r\n";
}

and here is my codeigniter code that is very slow... exporting data to CSV with 77000 rows took about 15 minutes excluding the download time..

public function exportCSV()
{
    set_time_limit(0);
    $delimiter = ",";
    $newline = "\r\n";

    $curr_date_time = date("l jS \of F Y h:i:s A");

    $this->products_model->set_venture($this->selected_venture['abbrev']);

    if($get_data = $this->input->get())
    {
        $data = $this->products_model->export_model($get_data);
        $download = $this->dbutil->csv_from_result($data, $delimiter, $newline);
        force_download('export - '.$curr_date_time.'.csv', $download); 
    }
    else
    {
        show_404('page', FALSE);
    }

}

public function export_model($params = NULL)
{
    if ($params != NULL)
    {
        if ($params['name_filter'] != '')
        {
            $this->crawler_db->like('name', $params['name_filter']);
        }

        if ($params['comp_filter'] != '')
        {
            $this->crawler_db->where('fk_competitor_website', $params['comp_filter']);
        }
    }

    return $this->crawler_db->get('pcrawler_'.$this->venture.'.products_final');

}

Answer 1

Hi check database util class can work for you its really simple to generate good CSV files here is the code

$this->load->dbutil();

$query = $this->db->query("SELECT * FROM mytable");

echo $this->dbutil->csv_from_result($query);

please read the document here Codeigniter CSV export with DB util

Answer 2

I would look at debugging the function, to see where it's actually slow.

Simple way is to use the Benchmarking Class to see if it's the query that's slow, or the call to csv_from_result().

Is it still slow when you don't pass any params to export_model()? If it's only slow when you run a like or where against the database, then maybe you need to add some indexes?

https://ellislab.com/codeigniter/user-guide/libraries/benchmark.html

Once you find the bottleneck, you can go from there to

Answer 3

code igniter's csv_from_result is really slow for me, i made a different one to suit my needs.. it can be re used for other CSV exports too...

public function array_to_csv($array)
{
   if (count($array) == 0) {
     return null;
   }
   ob_start();
   $df = fopen("php://output", 'w');
   fputcsv($df, array_keys(reset($array)));
   foreach ($array as $row) {
      fputcsv($df, $row);
   }
   fclose($df);
   return ob_get_clean();
}

usage:

$this->array_to_csv($array);