使用php从网站获取信息，以递归方式遍历每个HTML节点

Question

I have done lot research but I haven't found my answer. I am trying to get some information from webpage, Which have the following HTML structure

<div id="xxx" class="some1">
        <h1>This is the time</h1>
    <div class="ti12">
        <div class="sss"></div>
        <div class="sss">
                     <span class="hhh">
            <div class="sded">
                City:
                <span class="sh">CCC</span>
            </div>
                    </span>
        </div>
        </div>
        .
        .
    .
        <div class="pp12"></div>
</div>

Now, What i am doing is to fetch the NAME of the City and similarly other information in same way.

I have to find these information from above code.

$arr=array('City', 'Name', 'Address', 'DOB');

if exist fetch its value else leave it blank.

Hope my I am clear.

Following code it tried:

<?php
include "simple_html_dom.php";
   $html = new simple_html_dom();
   $listItem = array('City', 'Name', 'Address', 'DOB');
$html->load_file('simp.html');
$found=array();
    foreach($listItem as $item){
        $ret = $html->find('div[id=xxx] div',0);
        iterateParentNode($ret, $item);
        }
function iterateParentNode($ret1, $item1){
for ($node=0;$node < count($ret1->children());$node++){
    $child=$ret1->children($node);
    echo count($ret1->children())."<br/>";
    if(count($ret1->children())==1 && strpos($child, '<span class="sh"')!==false ){
        $found[$item1]=$ret1->find('span[class=sh]',0)->plaintext;
        return true;
    }else{
    goThroughChildNode($child, $item1);
    }
}
}
function goThroughChildNode($child1, $item2){
    echo $child1."ITEM:".$item2;
        if(strpos($child1, $item2)!==false){
            iterateParentNode($child1, $item2);
        }else{
            return false ;
        }
        return true;
}
foreach ($found as $structure=>$data){
  echo $structure."=>".$data."<br />";
}
?>

I know my PHP approach is not good, So please suggest me a good approach to do it with considering my PHP code.

Answer 1

It would probably be simplest to do this with a regex. Of course, it will break if the HTML structure changes.

if (ereg('<div.*?h1>(.*?)</h1>.*?City:.*?>(.*?)<', $input, $regs)) {
    $title = $regs[1];
    $city = $regs[2];
} else {
    $title = "";
    $city = "";
}

/*
Match 1 of 1
Matched text: <div id="xxx" class="some1">
        <h1>This is the time</h1>
    <div class="ti12">
        <div class="sss"></div>
        <div class="sss">
                     <span class="hhh">
            <div class="sded">
                City:
                <span class="sh">CCC<
Match offset: 0
Match length: 282
Group 1: This is the time
Group 1 offset: 42
Group 1 length: 16
Group 2: CCC
Group 2 offset: 278
Group 2 length: 3
*/

// <div.*?h1>(.*?)</h1>.*?City:.*?>(.*?)<
// 
// Match the characters "<div" literally «<div»
// Match any single character «.*?»
//    Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the characters "h1>" literally «h1>»
// Match the regular expression below and capture its match into backreference number 1 «(.*?)»
//    Match any single character «.*?»
//       Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the characters "</h1>" literally «</h1>»
// Match any single character «.*?»
//    Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the characters "City:" literally «City:»
// Match any single character «.*?»
//    Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the character ">" literally «>»
// Match the regular expression below and capture its match into backreference number 2 «(.*?)»
//    Match any single character «.*?»
//       Between zero and unlimited times, as few times as possible, expanding as needed (lazy) «*?»
// Match the character "<" literally «<»

Answer 2

One alternative to manual traversal is querying for the data instead. In DOMDocument this is commonly done with XPath, a language dedicated to exactly that job.

The library you use does not support XPath , however, PHP does support it out of the box. PHP also supports DOMDocument out of the box, so I think I can safely suggest you that as an alternative.

So in your case you are first looking into the the div with the ID:

//div[@id="xxx"]

and then inside a div in there somewhere:

   //div

and then you want another element in there if no specific name (children):

       //*

but those need to match a specific pattern: Here, containing a span with a class attribute having "sh", it must be the first span in there and before the span there must be some text:

            [
                span[@class="sh"]
                and span = span[@class="sh"]
                and span/preceding-sibling::text()
            ]

and of that child you want the first text node child:

                /text()[1]

So just to see this at a glance:

    //div[@id="xxx"]
        //div
            //*[
                span[@class="sh"]
                and span = span[@class="sh"]
                and span/preceding-sibling::text()
            ]
                /text()[1]

This will give you the named string like "City:" and so on. The next sibling (span) then will contain the value.

All you've got to do is wrap that into code (here I load a string, but you can also load a HTML file with loadHTMLFile() , check the DOMDocument link above for all the glory details):

$dom = new DOMDocument();
$dom->loadHTML($string);
$xp = new DOMXPath($dom);

foreach ($xp->query('
        //div[@id="xxx"]
            //div
                //*[
                    span[@class="sh"]
                    and span = span[@class="sh"]
                    and span/preceding-sibling::text()
                ]
                    /text()[1]
                '
         ) as $node
) {
    $name  = trim($node->nodeValue);
    $value = trim($node->nextSibling->nodeValue);
    printf("%s %s\n", $name, $value);
}

The output with your example HTML:

City: CCC

I hope this can motivate you to look into DOMDocument and helps you to explore the power of XPath.

Answer 3

It took me a while to get it right, but this code traverses the entire DOM with Simple HTML Dom. Hope someone can use it.

<?php
$html = new simple_html_dom();
$html->load('<html><body>'.$text.'</body></html>');

if(method_exists($html,"childNodes")){
    if($html->find('html')) {
    //IF NOT OK, THROW ERROR
}}

$e=$html->find('body',0);
$p=$e->childNodes(0);
if(!$p){
    //BODY HAS NO CHILDNODES< THROW ERROR
    }

    $loop=true;
    //SAFEGUARD, PREVENTS INDEFINITE LOOPS
    $i=$j=0; 
    $i_max=500;
    $j_max=500;


    while($loop==true){
        //SAFEGUARD, PREVENTS INDEFINITE LOOPS
        $i=0;$i++;if($i>$i_max){$loop=false;break;}

        //TEST IF NODE HAS CHILDREN
        $p=$e->childNodes(0);

        //NO CHILDREN
        if(!$p){
            //DO SOMETHING WITH NODE
            clean_dom($e->outertext);

            //TEST IF NODE HAS SIBLING
            $p=$e->next_sibling();
            if(!$p){
                //NO SIBLING
                    //TEST THE PARENT, LOOP TILL WE FIND A SIBLING
                    $j=0;$sib_loop=true;
                    while($sib_loop==true){
                        //SAFEGUARD, PREVENTS INDEFINITE LOOPS
                        $j++;if($j>$j_max){$sib_loop=false;break;}
                        //TEST IF THERE IS A PARENT
                        $e=$e->parent();
                            //NO PARENT, WE'VE REACHED THE TOP AGAIN
                            if(!$e){
                                echo'***THE END***';
                                $sib_loop=$loop=false;break;}
                            //ELSE, TEST IF PARENT HAS SIBLING
                            $p=$e->next_sibling();
                            //THERE IS A SIBBLING, GO THERE
                            if($p){
                                //DO SOMETHING WITH THIS NODE
                                clean_dom($e->outertext);
                                $e=$e->next_sibling();
                                $sib_loop=false;break;
                                }
                            else{
                                $ret=clean_dom($e->outertext,$all);
                                $e->outertext=$ret;
                                }

                        }       
                    }
            else{
                //GOTO SIBLING
                $e=$e->next_sibling();
                }
            }

        else{
        //THERE IS A CHILD
            $e=$e->childNodes(0);
            }
        }
$text=$html->save();
$html->clear(); 
unset($html);

function clean_dom($e){
    //DO SOMETHING HERE        
    }