无法使用Java下载文件

Question

I'm trying to download javascript libraries from highcharts.com. If I hit the url manually I'm able to see the file. But if I try it with my program, it returns 403 error. I'm able to download other js files but not highcharts. Here is my code.

try {


   String fileName = "highchartsjs"; //The file that will be saved on your computer
            URL link = new URL("http://code.highcharts.com/highcharts.js"); //The file that you want to download
            //Code to download
            InputStream in = new BufferedInputStream(link.openStream());
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            byte[] buf = new byte[1024];
            int n = 0;
            while (-1!=(n=in.read(buf)))
            {
                out.write(buf, 0, n);
            }
            out.close();
            in.close();
            byte[] response = out.toByteArray();

            FileOutputStream fos = new FileOutputStream(fileName);
            fos.write(response);
            fos.close();
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

Very strange error I'm facing. Let me know where I went wrong or they added a policy that we can't download it through program etc.., Thanks in anticipation.

Added Exception trace for reference:

java.io.IOException: Server returned HTTP response code: 403 for URL: http://code.highcharts.com/highcharts.js
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at java.net.URL.openStream(Unknown Source)
        at com.visualbi.readjson.SaveMap.downloadFiles(SaveMap.java:52)
        at com.visualbi.Boot.main(Boot.java:11)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.simontuffs.onejar.Boot.run(Boot.java:340)
        at com.simontuffs.onejar.Boot.main(Boot.java:166)

Answer 1

Use URLConnection and you should be able to access the requested web page from java by setting the user agent. The following should work:

        URL url = new URL("http://code.highcharts.com/highcharts.js");         
        URLConnection urlConn = url.openConnection();
        urlConn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.29 Safari/537.36");
        InputStream is = urlConn.getInputStream();
       //Do what you want to do with the InputStream

The error was the missing set of the User-Agent .

Answer 2

You could also use HttpClient: http://hc.apache.org/

HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
HttpResponse response = client.execute(request);
BufferedReader rd = new BufferedReader(new InputStreamReader(response
            .getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) {
    result.append(line);
}
FileOutputStream fos = new FileOutputStream(fileName);
fos.write(result.toString().getBytes());
fos.close();

Answer 3

I was facing a similar problem and was not resolved by setting user-agent property of URLConnection. But typecast to HttpURLConnection made it work.

Does not work. Gives 403.

URLConnection hc = url.openConnection();
hc.setRequestProperty("User-Agent", "curl/7.43.0");
FileUtils.copyURLToFile(url, temp);

Works

HttpURLConnection hc = (HttpURLConnection) url.openConnection();
hc.setRequestProperty("User-Agent", "curl/7.43.0");
FileUtils.copyURLToFile(url, temp);