站点地图生成器，从头开始构建

Question

I'd like to know how to build a site crawler, in php, that detects each page of a website and generates an entry in a xml file. I've seen plenty of websites doing this so I'm curious how to do it from scratch or there is any script or tutorial to teach that.

Answer 1

don't use regex. the proper way to parse html would be by using a DOMDocument object.

1. Load the first page into a DOMDocument object.
2. Use XPath statements to gather all of the anchor tag hrefs foudn in that page.
3. Use those values to find more pages to load, to start over with on step one again.

http://www.php.net/manual/en/class.domdocument.php

Answer 2

Here is the algorithm
Step 1-> Get a site's address, verify the address is in correct format and it ends with a page (www.xyz.com/page.html) not like (www.xyz.com/).
Step 2-> Get the contents of the file, using regular expression try to get the list of pages.
Step 3-> Harvest them in the DB for future use and do the step 2 on those files too.