php - how to handle cURL sending large size json data in POST efficiently?

Question

Here is the scenario. There is a method which fetches 156000 rows from DB table. I need to send this data to another php file which uses phpexcel to convert it into Excel file.

Below is the code of curl post request.

if($_SERVER['SERVER_NAME'] === 'localhost'){
        $domain = $_SERVER['REMOTE_ADDR'].':'.$_SERVER['SERVER_PORT'];
    }else{
        $domain = $_SERVER['SERVER_NAME'].':'.$_SERVER['SERVER_PORT'];
    }
    $prefix = $_SERVER['HTTPS'] ? 'https://' : 'http://';
    $relative   = '/rest/ExportExcelCsv.php';
    error_log($prefix.$domain.$relative);

    // converting array of returned rows into json
    $postData = json_encode($utilDataArray['tableData']);       

    $defaults = array(             
        CURLOPT_URL => $prefix.$domain.$relative, 
        CURLOPT_HEADER => 0, 
        CURLOPT_RETURNTRANSFER => TRUE, 
        CURLOPT_TIMEOUT => 500,
        CURLOPT_PROXY => "<proxy_url>",
        CURLOPT_PROXYPORT => 80,
        CURLOPT_POST => 1,
        CURLOPT_POSTFIELDS => array('data' => $postData)
    );

    $ch = curl_init(); 
    curl_setopt_array($ch, ($defaults));
    // curl_setopt($ch, CURLOPT_HTTPHEADER, array('Accept: application/json'));
    // curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));

    $result = curl_exec($ch);
    if( !$result ) 
    { 
        trigger_error(curl_error($ch)); 
    }
    var_dump($result);

    curl_close($ch);

In a statement, CURLOPT_POSTFIELDS => array('data' => $postData) , I am passing postData in an array with key data as ExportExcelCsv.php gets post data in that format with json_decode($_POST['data'], true) .

First issue is the size of "POST Content-Length". So, I've increased it with a statement ini_set('post_max_size', '100M'); ini_set('upload_max_filesize', '100M'); ini_set('post_max_size', '100M'); ini_set('upload_max_filesize', '100M'); in both php files. Still it didn't work. The size of post content is almost 42MB.

How do I send such large size json data in curl post request ? And how do to handle it efficiently ? Is there anyway to send data in chunk ? If yes, then how to handle that data while generating excel file out of it ?

Answer 1

First off, post_max_size and upload_max_filesize need to be in .htaccess, a vhost or in php.ini, not via ini_set . These, along with a few other ini values MUST be set before your script starts.

You can also probably also make the process a bit easier by first dumping your json data into a text file and sending the file via curl, rather than sending the data directly. A well written answer on how to send a file via curl can be found here: https://stackoverflow.com/a/15200804/4027341