How to scrape links from a a page with DOM & XPath?

Question

I have a page scraped with curl and am looking to grab all of the links with a certain id. As far as I can tell the best way to do this is with dom and xpath. The bellow code grabs a large number of the urls, but cuts many of them off and grabs text that is not a url.

$curl_scraped_page is the page scraped with curl.

$dom = new DOMDocument();
@$dom->loadHTML($curl_scraped_page);
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//a");

Am I on the right track? Do I just need to mess with the "/html/body//a" xpath syntax or do I need to add more to capture the id element?

Answer 1

You can also do it this way and you'll have onyl a tags which have an id and href :

$doc = new DOMDocument();
$doc->loadHTML($curl_scraped_page);
$xpath = new DOMXPath($doc);

$hrefs = $xpath->query('//a[@href][@id]');

Answer 2

I think that the easiest way is combining 2 following classes to pull information from another website:

Pull info from any HTML tag, contents or tag attribute: http://simplehtmldom.sourceforge.net/

Easy to handle curl, supports POST requests: https://github.com/php-curl-class/php-curl-class

Example:

include('path/to/curl.php');
include('path/to/simple_html_dom.php');
$url = 'http://www.example.com';

$curl = new Curl;
$html = str_get_html($curl->get($url)); //full HTML of website
$linksWithSpecificID = $html->find('a[id=foo]'); //returns array of elements

Check Simple HTML DOM Parser Manual from the upper link for the manipulation with HTML data.

Answer 3

$dom = new DOMDocument();
$dom->loadHTML($curl_scraped_page);

$links = $dom->getElementsByTagName('a');

$processed_links = array();

foreach ($links as $link)
{
    if ($link->hasAttribute('id') && $link->hasAttribute('href'))
    {
        $processed_links[$link->getAttribute('id')] = $link->getAttribute('href');
    }
}

This is the solution regarding your question.

Answer 4

http://simplehtmldom.sourceforge.net/

include('simple_html_dom.php');
$html = file_get_html('http://www.google.com/');
foreach($html->find('#www-core-css') as $e) echo $e->outertext . '<br>';