使用 Java 中的 HTTPClient 下載文件

Question

我正在嘗試編寫一個登錄網站，在搜索引擎中鍵入，獲取結果，然后下載從結果生成的 excel 文件的 Java 程序。 到目前為止，我可以正常登錄。 並發送搜索並獲取結果。 但是，我在下載 excel 文件時遇到了很多問題。

查看網站的源代碼，我在 excel 文件周圍看到了 Ajax 和 Javascript，所以我假設它是 ajax 幫助生成它。

<input id="toexcel" type="image" src="/websmart/v9.4/XLGP/images/Excel-icon.png" alt="To Excel" title="To Excel: Max 20000 Records" onclick="" />

JavaScript 部分：

$( document ).ready(function() {

        $('#toexcel').click(function(e) { 
            e.preventDefault();
            
            setTask('toexcel');
            
            var ajaxForm = $("#filter-form");
                        
            
            $(".spinner").show();

                var dataToSend = ajaxForm.serialize();
                $("#excelFrame").attr('src','V7BAE01R.pgm' + '?' + dataToSend);
            setTimeout(function() {
                            $(".spinner").hide();
                        }, 5000 );
                

使用 TamperData，當我單擊 Excel 文件導出時，它會發送一個發布請求（我設法在代碼的最后一部分發送），但我不確定從哪里獲取它。 我確實在篡改數據中看到了說 Application/vnd.ms-excel 的 Get

我不確定如何添加代碼以獲取 excel 文件。 下面，我嘗試使用 BufferReader，但它沒有獲取我的文件。 由於名稱值對，我簡化了一些代碼。

import java.util.List;
import java.util.ArrayList;
import org.apache.http.*;
import java.io.*;

import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.*;
import org.apache.http.impl.client.*;
import org.apache.http.message.*;
import org.apache.http.util.EntityUtils;
import org.apache.http.client.entity.*;
public class httpClientTest {
    
    
    
    public static void main (String[] args) throws ClientProtocolException, IOException {
            //Set up HttpClient
            CloseableHttpClient httpclient = HttpClients.createDefault();
            HttpGet httpGet = new HttpGet("http://website");
            CloseableHttpResponse response = httpclient.execute(httpGet);
            
            //Create Post request to log into the AS400 website
            HttpPost httpPost = new HttpPost("http://loginwebsite");
            
            List <NameValuePair> nvps = new ArrayList <NameValuePair>();
            
            nvps.add(new BasicNameValuePair("user","username"));
            nvps.add(new BasicNameValuePair("password","password"));
            nvps.add(new BasicNameValuePair("button", "Login"));
            nvps.add(new BasicNameValuePair("task", "extlogin"));
            httpPost.setEntity(new UrlEncodedFormEntity(nvps));
            response = httpclient.execute(httpPost);
            
            //Get Post response to ensure we logged in, which succeeds
            try{
                System.out.println(response.getStatusLine());   
                HttpEntity entity = response.getEntity();
                EntityUtils.consume(entity);
            } finally{
                response.close();
            }
            
            //Sent a Post request to filters out recoreds.
            httpPost = new HttpPost("http://searchresults");
            nvps.clear();
            nvps.add(new BasicNameValuePair("ActSts", "Edit"));
            nvps.add(new BasicNameValuePair("task", "filter"));
            nvps.add(new BasicNameValuePair("Field", "Plant"));
            response = httpclient.execute(httpPost);
            
            //Displays in printline the html/js of the page. This looks like it DOES display the search results
            //So it IS sending the Post request and receiving a response.
            BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent())); 
            String line = "";
            while ((line = rd.readLine()) != null) {
                System.out.println(line);
            }

        //try to buffer to read in.
        String link = "http://website.com/uri?ActSts=Edit&task=filter&Field=Plant";
        HttpGet get = new HttpGet(link);
        response = httpclient.execute(get);
        
        InputStream is = response.getEntity().getContent();
        String filePath = "C:\\Users\\WindowsUserName\\Downloads\\WODETAIL_List.xls";
        FileOutputStream fos = new FileOutputStream(new File(filePath));
        int inByte;
        while((inByte = is.read()) != -1)
            fos.write(inByte);
        is.close();
        fos.close();

我很確定我發布的數據是正確的，但我不確定如何獲取 excel 文件。 有人可以提供一些幫助嗎？

編輯我能夠下載一個文件，但它不是excel文件。 這是一個網頁，我認為這是一個小小的改進。 （之前，沒有下載任何東西，它只是掛在那里）問題是，我想我需要發送帶有此 get 請求的授權密鑰或 cookie 來下載文件。

編輯 2我發現如果我在登錄時在新選項卡中粘貼到http://website.com/uri?ActSts=Edit&task=filter&Field=Plant ，稍等片刻后，我得到了一個指向 excel 文件的鏈接. 所以最初我認為只要使用相同的 httpclient，HTTPClient 就會始終保持相同的 cookie，但顯然它沒有（？）我想我必須想辦法獲取 cookie 並發送它。

Answer 1

哦，天哪，我終於得到了一些有用的東西。 好的。 所以顯然 HTTPClient 在它開始出錯之前只能處理 2 個響應，根據這里： Why do me use HttpClients.createDefault() as HttpClient singleton instance execute third request always hang

因此，我將代碼更改為僅獲取登錄響應，然后獲取 excel 文件作為響應，然后退出。 我還添加了一些超時配置，並更改了先導出文件然后使用實體的順序。 我使用了單獨的第二個響應和第二個實體。 這似乎也有點幫助？ 我正在猜測。

import java.util.List;
import java.util.ArrayList;
import org.apache.http.*;
import java.io.*;

import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.CookieStore;
import org.apache.http.client.config.CookieSpecs;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.*;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.conn.ConnectionPoolTimeoutException;
import org.apache.http.cookie.Cookie;
import org.apache.http.impl.client.*;
import org.apache.http.message.*;
import org.apache.http.util.EntityUtils;
import org.apache.http.client.entity.*;

public class hcFeb {
    public static void main (String[] args) throws ClientProtocolException, IOException {
        //Set up Cookie settings and also Timeout settings
        CookieStore cookieStore = new BasicCookieStore();
        HttpClientContext context = HttpClientContext.create();
        context.setCookieStore(cookieStore);
        
        int CONNECTION_TIMEOUT = 80000;
        RequestConfig requestConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.DEFAULT)
                .setConnectionRequestTimeout(CONNECTION_TIMEOUT)
                .setConnectTimeout(CONNECTION_TIMEOUT)
                .setSocketTimeout(CONNECTION_TIMEOUT)
                .build();
        
        //Set up HttpClient
        CloseableHttpClient httpclient = HttpClients.custom().setDefaultRequestConfig(requestConfig).setDefaultCookieStore(cookieStore).disableContentCompression().build();
        
        HttpGet httpGet = new HttpGet("http://website");
        CloseableHttpResponse response = httpclient.execute(httpGet);
        
        //Create Post request to log into the website
        HttpPost httpPost = new HttpPost("http://loginwebsite");
        
        //Login to website
         List <NameValuePair> nvps = new ArrayList <NameValuePair>();

            nvps.add(new BasicNameValuePair("user","username"));
            nvps.add(new BasicNameValuePair("password","password"));
            nvps.add(new BasicNameValuePair("button", "Login"));
            nvps.add(new BasicNameValuePair("task", "extlogin"));
            httpPost.setEntity(new UrlEncodedFormEntity(nvps));
            response = httpclient.execute(httpPost);

        
        try{
            System.out.println(response.getStatusLine());   
            HttpEntity entity = response.getEntity();
            EntityUtils.consume(entity);                
        } finally{
        }
        
        //Send request for Excel file and download it.
        String link = "http://website.com/uri?ActSts=Edit&task=filter&Field=Plant";
        HttpGet get = new HttpGet(link);
        
        //maybe create new response
        HttpResponse response2;

        try{
            response2 = httpclient.execute(get,context);
            System.out.println(response2.getStatusLine());  
            HttpEntity entity1 = response2.getEntity();


            if (entity1 != null) {
                System.out.println("Entity isn't null");
                
                InputStream is = entity1.getContent();
                String filePath = "C:\\Users\\windowsUserName\\Downloads\\WODETAIL_List.xls";
                FileOutputStream fos = new FileOutputStream(new File(filePath));
                
                byte[] buffer = new byte[5600];
                int inByte;
                while((inByte = is.read(buffer)) > 0)
                    fos.write(buffer,0,inByte);
                is.close();
                fos.close();
                
                System.out.println("Excel File recieved");                  
                
                
                EntityUtils.toString(response2.getEntity());
                EntityUtils.consume(entity1);
                
            }
            
        } catch (ConnectionPoolTimeoutException e){
            //response.close();
            System.out.println(e.getMessage());
        } catch (IOException e){
            System.out.println(e.getMessage());
        }
        
    }
}