使用Apache HttpClient的多部分表单数据发布不起作用

Question

I'm trying to post to my website using a java desktop application. I used Apache HttpClient to make the post request and Jsoup to do some parsing after.

This is the form on my website:

<form name="item" action="http://example.com/index.php" method="post" enctype="multipart/form-data">
   <input type="hidden" name="CSRFName" value="CSRF1142289289_807321318">
   <input type="hidden" name="CSRFToken" value="2db57fc741cdb8c1b02a2f485508def7796162f5eebdd9f7d0ce9fd9aabb39ecd5df8c63a46560ae540ed9f26c40fdd33f0285e7b1f2ef18bc4b760ad3da4ae8">
   <fieldset>
      <input type="hidden" name="action" value="item_add_post">
      <input type="hidden" name="page" value="item">
      <input id="catId" type="hidden" name="catId" value="">
      <select id="select_1" name="select_1" depth="1" class="valid">
         <option value="0">Selectează categoria</option>
         <option value="108">Cazare si Pensiuni</option>
         <option value="105">Transport si Turism</option>
      </select>
      <input id="titlero_RO" type="text" name="title[ro_RO]" value="">
      <textarea id="descriptionro_RO" name="description[ro_RO]" rows="10" class="uniform"></textarea>
      <input id="price" type="text" name="price" value="">           
      <select name="currency" id="currency">
      <option value="EUR">euro</option>
         <option value="RON" selected="selected">lei</option>
         <option value="USD">dollar</option>
      </select>
      <input type="file" name="photos[]"><span class="filename" style="-webkit-user-select: none;">
      <select name="regionId" id="regionId">
         <option value="">Regiunea</option>
         <option value="782153">Alba</option>
      </select>
      <select name="cityId" id="cityId" disabled="disabled">
         <option value="">Orasul</option>
         <option value="462667">Abrud</option>
      </select>
      <input id="cityArea" type="text" name="cityArea" value="">
      <input id="cityAreaId" type="hidden" name="cityAreaId" value="">
      <input id="contactName" type="text" name="contactName" value="">                       
      <input id="contactEmail" type="text" name="contactEmail" value="">            
      <div class="button" style="-webkit-user-select: none;"><span>Publică<button type="submit">Publică</button></span></div>
   </fieldset>
</form>

Of course i've removed some stuff from it such as divs, css for this, I kept the minimum.

If I submit the form through my browser the actual data sent is this one:

Remote Address:xx.xxx.xx.xxx:80
Request URL:http://example.com.ro/index.php
Request Method:POST
Status Code:302 Moved Temporarily
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate,lzma,sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:2398
Content-Type:multipart/form-data; boundary=----WebKitFormBoundarybWBRwGAP2qS2vOYX
Cookie:_ugtopperm=1; b620f=last_submit_item%261406384018; _ugtopsecure=1;         osclass=f8e8a642a5d853ef3ca332a44604b975
Host:example.com.ro
Origin:http://example.com.ro
Referer:http://example.com.ro/adauga
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)     Chrome/36.0.1985.125 Safari/537.36 OPR/23.0.1522.60
Request Payload
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="CSRFName"

CSRF1142289289_807321318
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="CSRFToken"

2db57fc741cdb8c1b02a2f485508def7796162f5eebdd9f7d0ce9fd9aabb39ecd5df8c63a46560ae540ed9f26c40fdd33f0285e7b1f2ef18bc4b760ad3da4ae8
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="action"

item_add_post
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="page"

item
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="catId"

98
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="select_1"

98
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="title[ro_RO]"

This is the ad title
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="description[ro_RO]"

This is the ad description lots of text etc...
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="price"

5
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="currency"

RON
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="photos[]"; filename=""
Content-Type: application/octet-stream


------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="regionId"

782162
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="cityId"

463357
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="cityArea"

0123457617
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="cityAreaId"


------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="contactName"

nic
------WebKitFormBoundarybWBRwGAP2qS2vOYX
Content-Disposition: form-data; name="contactEmail"

nicolae_aldea@yahoo.com
------WebKitFormBoundarybWBRwGAP2qS2vOYX--
Response Headersview source
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:Keep-Alive
Content-Length:0
Content-Type:text/html
Date:Wed, 30 Jul 2014 13:41:38 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Keep-Alive:timeout=3, max=100
Location:http://example.com.ro/prestari-servicii
Pragma:no-cache
Server:Apache/2.2.27 (Unix) mod_ssl/2.2.27 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4
Set-Cookie:b620f=last_submit_item%261406727698; expires=Thu, 30-Jul-2015 13:41:38 GMT;     path=/
X-Powered-By:PHP/5.3.28

And it works ok. After I click the Publish button (in the browser) I get redirected from /adauga to /prestari-servicii where I can see that ad I just typed.

Now I tried to mimic the same thing in java using Apache HttpClient

package abcd;

import java.io.File;
import java.util.List;
import java.util.Random;

import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.cookie.Cookie;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.entity.mime.content.StringBody;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class Test {
    public static void main(String[] args) throws Exception {
        BasicCookieStore cookieStore = new BasicCookieStore();
        CloseableHttpClient client = HttpClients.custom()
            .setDefaultCookieStore(cookieStore).build();

        //trying to get the cookies if any?!
        HttpGet get = new HttpGet("http://example.com.ro/adauga");
        get.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
        get.setHeader("Accept-Encoding", "gzip,deflate,lzma,sdch");
        get.setHeader("Accept-Language", "en-US,en;q=0.8");
        get.setHeader("Cache-Control", "max-age=0");
        get.setHeader("Connection", "keep-alive");
    /* HERE IS THE PART I THINK IT MESSES UP
        get.setHeader("Cookies", what should i set here? is the job done by client
    having the cookieStore set?? 
    */
        get.setHeader("Host", "example.com.ro");
        get.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36 OPR/23.0.1522.60");
        CloseableHttpResponse response1 = client.execute(get);

        List<Cookie> cookies = cookieStore.getCookies();
        if (cookies.isEmpty()) {
            System.out.println("None");
        } else {
            for (int i = 0; i < cookies.size(); i++) {
                System.out.println("- " + cookies.get(i).toString());
            }
        }

        response1.close();

        //my site uses CSRF protection in the form so first I take these using Jsoup
        Document html = Jsoup.connect("http://example.com.ro/adauga").get();
        String CSRFName = html.getElementsByAttributeValue("name", "CSRFName").get(0).val();
        String CSRFToken = html.getElementsByAttributeValue("name", "CSRFToken").get(0).val();

        File image = new File("C:\\Users\\DESKTOP\\Desktop\\Capture.PNG");
        FileBody photos = new FileBody(image, ContentType.APPLICATION_OCTET_STREAM);

        StringBody csrfName = new StringBody(CSRFName, ContentType.MULTIPART_FORM_DATA);
        StringBody csrfToken = new StringBody(CSRFToken, ContentType.MULTIPART_FORM_DATA);
        StringBody action = new StringBody("item_add_post", ContentType.MULTIPART_FORM_DATA);
        StringBody page = new StringBody("item", ContentType.MULTIPART_FORM_DATA);
        StringBody catId = new StringBody("98", ContentType.MULTIPART_FORM_DATA);
        StringBody select_1 = new StringBody("98", ContentType.MULTIPART_FORM_DATA);
        StringBody title = new StringBody("Sample title text etc", ContentType.MULTIPART_FORM_DATA);
        StringBody description = new StringBody("Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.", ContentType.MULTIPART_FORM_DATA);
        StringBody price = new StringBody("2", ContentType.MULTIPART_FORM_DATA);
        StringBody currency = new StringBody("RON", ContentType.MULTIPART_FORM_DATA);
        StringBody regionId = new StringBody("782162", ContentType.MULTIPART_FORM_DATA);
        StringBody cityId = new StringBody("463357", ContentType.MULTIPART_FORM_DATA);
        StringBody cityArea = new StringBody("0123577617", ContentType.MULTIPART_FORM_DATA);
        StringBody cityAreaId = new StringBody("", ContentType.MULTIPART_FORM_DATA);
        StringBody contactName = new StringBody("Garry", ContentType.MULTIPART_FORM_DATA);
        StringBody contactEmail = new StringBody("bogus@email.abc", ContentType.MULTIPART_FORM_DATA);

        HttpPost post = new HttpPost("http://example.com.ro/index.php");
        post.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
        post.setHeader("Accept-Encoding", "gzip,deflate,lzma,sdch");
        post.setHeader("Accept-Language", "en-US,en;q=0.8");
        post.setHeader("Cache-Control", "max-age=0");
        post.setHeader("Connection", "keep-alive");
        post.setHeader("Content-Type", "multipart/form-data");
    //again, I think i'm doing something wrong here. Am I supposed to set
    //boundary as this??
        post.setHeader("boundary", generateBoundary());
    //should i set cookies after? if so, how?!
        post.setHeader("Host", "example.com.ro");
        post.setHeader("Origin", "http://example.com.ro");
        post.setHeader("Referer", "http://example.com.ro/adauga");
        post.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36 OPR/23.0.1522.60");

        HttpEntity entity = MultipartEntityBuilder.create()
                .addPart("CSRFName", csrfName)
                .addPart("CSRFToken", csrfToken)
                .addPart("action", action)
                .addPart("page", page)
                .addPart("catId", catId)
                .addPart("select_1", select_1)
                .addPart("title[ro_RO]", title)
                .addPart("description[ro_RO]", description)
                .addPart("price", price)
                .addPart("currency", currency)
                .addPart("photos[]", photos)
                .addPart("regionId", regionId)
                .addPart("cityId", cityId)
                .addPart("cityArea", cityArea)
                .addPart("cityAreaId", cityAreaId)
                .addPart("contactName", contactName)
                .addPart("contactEmail", contactEmail)
                .build();


        post.setEntity(entity);


        CloseableHttpResponse response = client.execute(post);
        System.out.println("After executing the post request... :" + response.getStatusLine().getStatusCode() + " - " + response.getStatusLine());

        String htmlPageResponse = EntityUtils.toString(response.getEntity());
        System.out.println();
        System.out.println(htmlPageResponse);
        response.close();
        client.close();
    }

    protected static String generateBoundary() {
        StringBuilder buffer = new StringBuilder();
        Random rand = new Random();
        int count = rand.nextInt(11) + 30; // a random size from 30 to 40
        for (int i = 0; i < count; i++) {
        buffer.append(MULTIPART_CHARS[rand.nextInt(MULTIPART_CHARS.length)]);
        }
        return buffer.toString();
   }

    private final static char[] MULTIPART_CHARS =
        "-_1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
         .toCharArray();
}

After I run this I get 200 - HTTP/1.1 200 OK Instead of 302 as I get when I use the browser. I also checked the html response and it was the original index.php page instead of the redirect where I was supposed to view my just entered message.

So did I set the cookies right? Or I messed something in the request headers? Like the boundary etc. I had this problem for a week almost. I tried every single httpclient multipart example but they don't seem to work with a "real world" website.

Thank you!

Answer 1

Have you tried to compare the request you created using Apache HttpClient with the one you posted above? Doing so, you could easily check if the two are really the same, and whether you did something wrong or not in setting the cookies.

To view the request before executing it you can print it before executing it. Try System.out.println(get.toString()); or alternatively, you can use HttpMessage's method - getFirstHeader("cookie") to look into the Header you set.

But in order to mimic the actual request, I guess you should create a Cookie object and then use methods from SetCookie Interface to set it accordingly to Cookie:_ugtopperm=1; b620f=last_submit_item%261406384018; _ugtopsecure=1; Cookie:_ugtopperm=1; b620f=last_submit_item%261406384018; _ugtopsecure=1; . At that point, you can use cookieStore.addCookie(Cookie cookie) method just after BasicCookieStore cookieStore = new BasicCookieStore();

I haven't never done this myself. But this is what I understood by taking a look to the APIs. I hope this might be of some help.