I'm trying to download javascript libraries from highcharts.com. If I hit the url manually I'm able to see the file. But if I try it with my program, it returns 403 error. I'm able to download other js files but not highcharts. Here is my code.
try {
String fileName = "highchartsjs"; //The file that will be saved on your computer
URL link = new URL("http://code.highcharts.com/highcharts.js"); //The file that you want to download
//Code to download
InputStream in = new BufferedInputStream(link.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
int n = 0;
while (-1!=(n=in.read(buf)))
{
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
FileOutputStream fos = new FileOutputStream(fileName);
fos.write(response);
fos.close();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
Very strange error I'm facing. Let me know where I went wrong or they added a policy that we can't download it through program etc.., Thanks in anticipation.
Added Exception trace for reference:
java.io.IOException: Server returned HTTP response code: 403 for URL: http://code.highcharts.com/highcharts.js
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at com.visualbi.readjson.SaveMap.downloadFiles(SaveMap.java:52)
at com.visualbi.Boot.main(Boot.java:11)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.simontuffs.onejar.Boot.run(Boot.java:340)
at com.simontuffs.onejar.Boot.main(Boot.java:166)
Use URLConnection and you should be able to access the requested web page from java by setting the user agent. The following should work:
URL url = new URL("http://code.highcharts.com/highcharts.js");
URLConnection urlConn = url.openConnection();
urlConn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.29 Safari/537.36");
InputStream is = urlConn.getInputStream();
//Do what you want to do with the InputStream
The error was the missing set of the User-Agent
.
You could also use HttpClient: http://hc.apache.org/
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(url);
HttpResponse response = client.execute(request);
BufferedReader rd = new BufferedReader(new InputStreamReader(response
.getEntity().getContent()));
StringBuffer result = new StringBuffer();
String line = "";
while ((line = rd.readLine()) != null) {
result.append(line);
}
FileOutputStream fos = new FileOutputStream(fileName);
fos.write(result.toString().getBytes());
fos.close();
I was facing a similar problem and was not resolved by setting user-agent property of URLConnection. But typecast to HttpURLConnection made it work.
Does not work. Gives 403.
URLConnection hc = url.openConnection();
hc.setRequestProperty("User-Agent", "curl/7.43.0");
FileUtils.copyURLToFile(url, temp);
Works
HttpURLConnection hc = (HttpURLConnection) url.openConnection();
hc.setRequestProperty("User-Agent", "curl/7.43.0");
FileUtils.copyURLToFile(url, temp);
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.