[英]Download a file with HTTPClient in Java
I'm trying to write a java program that logs into a website, types in a search engine, gets the result, and then downloads the excel file that is generated from the results.我正在尝试编写一个登录网站,在搜索引擎中键入,获取结果,然后下载从结果生成的 excel 文件的 Java 程序。 So far, I can log in ok.
到目前为止,我可以正常登录。 and send a search in and get the results.
并发送搜索并获取结果。 However, I'm having a lot of problems downloading the excel file.
但是,我在下载 excel 文件时遇到了很多问题。
Looking at the website's source code, I see Ajax and Javascript around the excel file, so I'm assuming it's ajax that helps produce it.查看网站的源代码,我在 excel 文件周围看到了 Ajax 和 Javascript,所以我假设它是 ajax 帮助生成它。
<input id="toexcel" type="image" src="/websmart/v9.4/XLGP/images/Excel-icon.png" alt="To Excel" title="To Excel: Max 20000 Records" onclick="" />
The JavaScript part: JavaScript 部分:
$( document ).ready(function() {
$('#toexcel').click(function(e) {
e.preventDefault();
setTask('toexcel');
var ajaxForm = $("#filter-form");
$(".spinner").show();
var dataToSend = ajaxForm.serialize();
$("#excelFrame").attr('src','V7BAE01R.pgm' + '?' + dataToSend);
setTimeout(function() {
$(".spinner").hide();
}, 5000 );
Using TamperData, when I click the Excel File Export, it sends a post request (which I manage to send in the last part of the code) but I'm not sure where to Get it.使用 TamperData,当我单击 Excel 文件导出时,它会发送一个发布请求(我设法在代码的最后一部分发送),但我不确定从哪里获取它。 I do see in tamperdata the Get that says Application/vnd.ms-excel
我确实在篡改数据中看到了说 Application/vnd.ms-excel 的 Get
I'm not sure what to do to add in the code to get the excel file.我不确定如何添加代码以获取 excel 文件。 Below, I tried to use BufferReader, but it doesn't get my file.
下面,我尝试使用 BufferReader,但它没有获取我的文件。 Some of the code I simplified because of the Name Value pairs.
由于名称值对,我简化了一些代码。
import java.util.List;
import java.util.ArrayList;
import org.apache.http.*;
import java.io.*;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.*;
import org.apache.http.impl.client.*;
import org.apache.http.message.*;
import org.apache.http.util.EntityUtils;
import org.apache.http.client.entity.*;
public class httpClientTest {
public static void main (String[] args) throws ClientProtocolException, IOException {
//Set up HttpClient
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://website");
CloseableHttpResponse response = httpclient.execute(httpGet);
//Create Post request to log into the AS400 website
HttpPost httpPost = new HttpPost("http://loginwebsite");
List <NameValuePair> nvps = new ArrayList <NameValuePair>();
nvps.add(new BasicNameValuePair("user","username"));
nvps.add(new BasicNameValuePair("password","password"));
nvps.add(new BasicNameValuePair("button", "Login"));
nvps.add(new BasicNameValuePair("task", "extlogin"));
httpPost.setEntity(new UrlEncodedFormEntity(nvps));
response = httpclient.execute(httpPost);
//Get Post response to ensure we logged in, which succeeds
try{
System.out.println(response.getStatusLine());
HttpEntity entity = response.getEntity();
EntityUtils.consume(entity);
} finally{
response.close();
}
//Sent a Post request to filters out recoreds.
httpPost = new HttpPost("http://searchresults");
nvps.clear();
nvps.add(new BasicNameValuePair("ActSts", "Edit"));
nvps.add(new BasicNameValuePair("task", "filter"));
nvps.add(new BasicNameValuePair("Field", "Plant"));
response = httpclient.execute(httpPost);
//Displays in printline the html/js of the page. This looks like it DOES display the search results
//So it IS sending the Post request and receiving a response.
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
String line = "";
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
//try to buffer to read in.
String link = "http://website.com/uri?ActSts=Edit&task=filter&Field=Plant";
HttpGet get = new HttpGet(link);
response = httpclient.execute(get);
InputStream is = response.getEntity().getContent();
String filePath = "C:\\Users\\WindowsUserName\\Downloads\\WODETAIL_List.xls";
FileOutputStream fos = new FileOutputStream(new File(filePath));
int inByte;
while((inByte = is.read()) != -1)
fos.write(inByte);
is.close();
fos.close();
I'm pretty sure I'm Posting the data right, but I'm not sure about how to Get the excel file.我很确定我发布的数据是正确的,但我不确定如何获取 excel 文件。 Could anybody offer some help?
有人可以提供一些帮助吗?
Edit I was able to download a file, but it wasn't the excel file.编辑我能够下载一个文件,但它不是excel文件。 It was a webpage, and I think it's a little bit of an improvement.
这是一个网页,我认为这是一个小小的改进。 (Before, nothing downloaded, it just hanged there) The problem was, I think I need to send an authorization key or a cookie with this get request to download the file.
(之前,没有下载任何东西,它只是挂在那里)问题是,我想我需要发送带有此 get 请求的授权密钥或 cookie 来下载文件。
Edit 2 I've discovered if I just paste to http://website.com/uri?ActSts=Edit&task=filter&Field=Plant in a new tab while logged in, after waiting a little while, I get a link to the excel file.编辑 2我发现如果我在登录时在新选项卡中粘贴到http://website.com/uri?ActSts=Edit&task=filter&Field=Plant ,稍等片刻后,我得到了一个指向 excel 文件的链接. So originally I thought HTTPClient maintains the same cookies throughout as long as the same httpclient is used but apparently it doesn't(?) I guess I have to figure out a way to get a cookie and send it.
所以最初我认为只要使用相同的 httpclient,HTTPClient 就会始终保持相同的 cookie,但显然它没有(?)我想我必须想办法获取 cookie 并发送它。
Oh my god I finally got something that worked.哦,天哪,我终于得到了一些有用的东西。 Ok.
好的。 So apparently HTTPClient can only handle 2 responses before it starts to bug out, according to here: Why does me use HttpClients.createDefault() as HttpClient singleton instance execute third request always hang
所以显然 HTTPClient 在它开始出错之前只能处理 2 个响应,根据这里: Why do me use HttpClients.createDefault() as HttpClient singleton instance execute third request always hang
So instead, I changed my code to just get a login response, then get the excel file as a response and then quit.因此,我将代码更改为仅获取登录响应,然后获取 excel 文件作为响应,然后退出。 I also added some timeout configurations and also changed order from Exporting the file first and then Consuming the entity.
我还添加了一些超时配置,并更改了先导出文件然后使用实体的顺序。 I used a separate 2nd response and 2nd entity.
我使用了单独的第二个响应和第二个实体。 That seemed to have helped a bit too?
这似乎也有点帮助? I'm guessing.
我正在猜测。
import java.util.List;
import java.util.ArrayList;
import org.apache.http.*;
import java.io.*;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.CookieStore;
import org.apache.http.client.config.CookieSpecs;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.*;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.conn.ConnectionPoolTimeoutException;
import org.apache.http.cookie.Cookie;
import org.apache.http.impl.client.*;
import org.apache.http.message.*;
import org.apache.http.util.EntityUtils;
import org.apache.http.client.entity.*;
public class hcFeb {
public static void main (String[] args) throws ClientProtocolException, IOException {
//Set up Cookie settings and also Timeout settings
CookieStore cookieStore = new BasicCookieStore();
HttpClientContext context = HttpClientContext.create();
context.setCookieStore(cookieStore);
int CONNECTION_TIMEOUT = 80000;
RequestConfig requestConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.DEFAULT)
.setConnectionRequestTimeout(CONNECTION_TIMEOUT)
.setConnectTimeout(CONNECTION_TIMEOUT)
.setSocketTimeout(CONNECTION_TIMEOUT)
.build();
//Set up HttpClient
CloseableHttpClient httpclient = HttpClients.custom().setDefaultRequestConfig(requestConfig).setDefaultCookieStore(cookieStore).disableContentCompression().build();
HttpGet httpGet = new HttpGet("http://website");
CloseableHttpResponse response = httpclient.execute(httpGet);
//Create Post request to log into the website
HttpPost httpPost = new HttpPost("http://loginwebsite");
//Login to website
List <NameValuePair> nvps = new ArrayList <NameValuePair>();
nvps.add(new BasicNameValuePair("user","username"));
nvps.add(new BasicNameValuePair("password","password"));
nvps.add(new BasicNameValuePair("button", "Login"));
nvps.add(new BasicNameValuePair("task", "extlogin"));
httpPost.setEntity(new UrlEncodedFormEntity(nvps));
response = httpclient.execute(httpPost);
try{
System.out.println(response.getStatusLine());
HttpEntity entity = response.getEntity();
EntityUtils.consume(entity);
} finally{
}
//Send request for Excel file and download it.
String link = "http://website.com/uri?ActSts=Edit&task=filter&Field=Plant";
HttpGet get = new HttpGet(link);
//maybe create new response
HttpResponse response2;
try{
response2 = httpclient.execute(get,context);
System.out.println(response2.getStatusLine());
HttpEntity entity1 = response2.getEntity();
if (entity1 != null) {
System.out.println("Entity isn't null");
InputStream is = entity1.getContent();
String filePath = "C:\\Users\\windowsUserName\\Downloads\\WODETAIL_List.xls";
FileOutputStream fos = new FileOutputStream(new File(filePath));
byte[] buffer = new byte[5600];
int inByte;
while((inByte = is.read(buffer)) > 0)
fos.write(buffer,0,inByte);
is.close();
fos.close();
System.out.println("Excel File recieved");
EntityUtils.toString(response2.getEntity());
EntityUtils.consume(entity1);
}
} catch (ConnectionPoolTimeoutException e){
//response.close();
System.out.println(e.getMessage());
} catch (IOException e){
System.out.println(e.getMessage());
}
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.