PHP Curl: Getting a directory listing and downloading directories connecting to HTTP

Question

I'm fairly new to CURL and I was able to fetch individual files like this:

$c_session = curl_init();

curl_setopt ($c_session, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($c_session, CURLOPT_URL, $uri);
curl_setopt ($c_session, CURLOPT_TIMEOUT, '12');

$content = curl_exec($c_session);
curl_close ($c_session);

Now I need to be able to list directories and download them using CURL. The catch is I need to connect to an HTTP server and not an FTP one.

Answer 1

I don't think there is a native directory listing function in HTTP. The closest you'll get is the DirectoryListing that Apache and other Web servers may produce when accessing a folder URL. But that is HTML and you'll have to parse it first.

Better use FTP or if you can, have a server side script generate a simple list that you can download, parse and process.

Answer 2

You're going to have to parse a list generated by the server, whether that is by DirectoryListing as above, or another server-side script that generates a list of links.

You'll then parse the HTML and pull out of all the a href tags.

If you're relying on the output of another script (Directorylisting), you may need to run the HTML through tidy to produce XHTML, then pass that into simplexml. You can then write an xpath query like '//a' and retrieve all the attributes.

$list = array();
$x = new SimpleXMLElement($stringfromcurl);
foreach ($x->xpath('//a') as $node) {
    curl_fetch_href($x['href']);
}

Or... generate the list yourself as something a little easier to parse, then do the same sort of deal.

This is the equivalent of doing something like wget -r -l1

Answer 3

When you go to a host/path/ If there is no index.html many servers will list the names and links to files. Not all servers are configured to show empty directories. If the server you are connecting to is , you need to screen scrape the generated directory listing.

Take a look at the simple_html_dom parsing library for this.

Answer 4

Is the server using WebDAV ? If so, you may be able to find a library for PHP that can allow you to do this. Chances are slim, however, especially if you're trying to get stuff from a public-facing web server.