使用Java从网站下载多个文件

Question

I am logging onto a secure site using proxy and want to be able to download all the files and folders onto my local disc. This is what i have so far.

EDIT -**Currently the code below will start at a given root directory and download all files in all subdirectories ... pretty cool :) but it doesnt duplicate the folder structure which is what i need. Any help please? **EDIT

First of all i get 4 arguments (so can be used on cmd line on Linux)

1) url of directory i want to download 2) username of secure login 3) psw 4) directory where do i want the files saved on my local disc

       public class ApacheUrl4
{
// this is the entry point for what I want the instase of the class to do
    public static void main(String args[]) throws Exception {

        String url  = args[0];
        final String username  = args[1];
        final String password1  = args[2];
        String directory  = args[3];

        checkArguments(args);

        ApacheUrl4 max = new ApacheUrl4();
        max.process(url, username, password1, directory);

    }
    public void process (String url, String username1, String password1, String directory) throws Exception {

        final char[] password  = password1.toCharArray();   
        final String username = username1;
         Authenticator.setDefault(new Authenticator(){
              protected  PasswordAuthentication  getPasswordAuthentication(){
               PasswordAuthentication p=new PasswordAuthentication(username , password);
               return p;
              }
             });


        BufferedInputStream in = null;
        BufferedInputStream in2 = null;
        FileOutputStream fout = null;
    // proxy 
        String proxyip = "000.000.000" ;
        int proxyport = 8080;
        Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyip, proxyport));
     // URL connection to file 
        URL file = new URL(url);
        URLConnection connection = file.openConnection(proxy);      
        ((HttpURLConnection)connection).getResponseCode();
        int reponsecode = ((HttpURLConnection)connection).getResponseCode();
        System.out.println("response code " + reponsecode);


        if (reponsecode == HttpURLConnection.HTTP_FORBIDDEN){
            System.out.println("Invalid username or psw");
            return;
        }
        if (reponsecode != HttpURLConnection.HTTP_OK){
            System.out.println("Unable to find response");
            return;
        }





        //Save the file into the chosen folder
        in = new BufferedInputStream(connection.getInputStream());

        //Create instance of DocumentBuilderFactory
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        //Get the DocumentBuilder
        DocumentBuilder docBuilder = factory.newDocumentBuilder();
        //Using existing XML Document
        Document doc = docBuilder.parse(in);

        //create the root element 
        Element root = doc.getDocumentElement();
        NodeList nodeList = root.getElementsByTagName("li"); 


        for(int i=0; i<nodeList.getLength(); i++){
          Node childNode = nodeList.item(i);
          if (childNode.getTextContent().contains("/")) {


            //  System.out.println(url + childNode.getTextContent());
                process(url + childNode.getTextContent(), username, password1, directory);                        

        }

    if (childNode.getTextContent().contains(".") && !childNode.getTextContent().contains("..")) {


            String textcon =  url + childNode.getTextContent();
            System.out.println("aaa " + textcon);

            if (url.endsWith("/")) {
                System.out.println("ends with a /");    
            }

            textcon = textcon.replace( " ", "%20");
            URL file2 = new URL(textcon);

            String[] urlparts = textcon.split("/");
            int urllength = urlparts.length;
            String lastarray = urlparts[urllength-2];
            System.out.println("last array " + lastarray);


            URLConnection connection2 = file2.openConnection(proxy);        
            in2 = new BufferedInputStream(connection2.getInputStream());
            String test2 = childNode.getTextContent();
            System.out.println("eeee " + childNode.getTextContent());

            String filename = (directory + test2 );
              File f=new File(filename);
                  if(f.isDirectory())
                  continue;





              //InputStream inputStream= new FileInputStream("InputStreamToFile.java");
              OutputStream out=new FileOutputStream(f);
              byte buf[]=new byte[1024];
              int len;
              while((len=in2.read(buf))>0)
              out.write(buf,0,len);
              out.close();
              in2.close();


        }
    }
}




    // this is part of the validation of arguments provided by user
    private static void checkArguments(String[] args) {
        while (args.length < 4 || args[0].isEmpty() || args.length > 4 ) {
                System.out.println("Please specify five arguments in the following format \n "  +
                " URL USERNAME PASWORD FILEPATH FILENAME " +
                "EG: \"java helloW http://www.google.com user_name password C:\\path/dir/ filename.exe\" ");
                System.exit(1);
         }
    }
}

Answer 1

In order to download the files in the directory, you first need the directory listing. This is generated automatically by the server, if it is allowed. First, use your browser to check if this is the case on this specific server.

Then you will need to parse the listing page, and download each url. The bad news is that there is no standard for these pages. The good news is that most of the internet is hosted on apache or IIS, so if you can manage these two, you've got a good part covered.

You could probably get away with just parsing the file as xml (xhtml) and using xpath to recover all the urls.