简体   繁体   中英

PHP CURL: Crawling multiples pages in a loop

I am trying to get data from multiple urls by loop. In the first loop I am getting the data, but after that "No DATA RECEIVED" message is displayed by the browser. I tried to delay the function but it doesn't work. What should I do? I always get the first page, but after that I am not getting the data. Here is my code:

public function Crawl(){

    $url=array(0=>array("ABCD"=>
        array(0=>"http://abcd.com"
        )),
        1=>array("XYZ"=>
        array(0=>"xyz.com"
        ))
        );

        foreach ($url as $key => $value) {                
            $this->Cron($value);                
            sleep(5);
        }

}


public function Cron($url=null){        

    ob_start();
    set_time_out(0);
    foreach($url as $urlkey=>$urlvalue){

         for($prodcount=0;$prodcount<count($urlvalue);$prodcount++){

         $ch = curl_init();  // Initialising cURL
        curl_setopt($ch, CURLOPT_URL, $urlvalue[$prodcount]);    // Setting cURL's URL option with the $url variable passed into the function
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); // Setting cURL's option to return the webpage data
        $data = curl_exec($ch); // Executing the cURL request and assigning the returned data to the $data variable
        curl_close($ch);    // Closing cURL
         //echo $data;   // Returning the data from the function

        $html = str_get_html($data);

        echo $html;

         ob_flush();
    }

}
}

I think it could be a problem that the html is not well formed.

Try adding tags and the most important:

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM