How in Java to submit html logon form that has hidden csrf_token_login field with generated value

Question

I need to automate file downloading from a website. The file download button appears only after login for which I was provided username and password. In login form there are two more hidden fields one of which is csrf_token_login with a generated value:

<input type="hidden" name="csrf_token_login" value="nl9YERDFpecfITb8QwFWneoaefykxp2b" />

It is clear how to code this in Java (using java.net.HttpUrlConnection) if I would have just login and password (there is excellent explanation for this in Using java.net.URLConnection to fire and handle HTTP requests ): submit POST request, get cookies and set them for any subsequent request. But how can I get a generated value of csrf_token_login on the login form and submit it with other values?

Reading it using getInputStream() on the HttpURLConnection of a login page gives me the csrf value. But at the same time this establishes connection and prevents from setting connection properties for posting data:

private HttpURLConnection logUrlCon;
... 
BufferedReader logInput = new BufferedReader(new InputStreamReader(logUrlCon.getInputStream())); 
... // read and get csrf value OK

logUrlCon.setDoOutput(true); // throws java.lang.IllegalStateException: Already connected

Is there any way of getting this csrf_tiken_login value generated in a login form AND posting it with username and password?

Answer 1

Read login page content and extract the data using regular expressions. Your hidden field has a very distinctive form (with a unique name, etc.), so perfectly suitable for regular expression based data extraction.