How to write a java program that can post url on browser and log the results from html div or request response of HTTP?

Question

I am planning to write a java program where I have the url of website x with which I will appending number from 1 to 100 and I will be getting result from the website.

Should I write using request and response of HTTP or mere java program where the url as string would do? If I am getting the result as posted on browser, how to get the values from a div and write it to a text file. I guess the other option is also to get it via response.

Answer 1

All you need is a programatic Browser, which submits the request and gets you the response,

You can study the Http Request and Response Objects under Tcp/Ip Protocol stack and implement your own, but instead of Reinventing the wheel, you can use the apache commons Http Components Project, which has all this implemented Apache Http Components

Answer 2

I'm not sure if you will be able to control the browser using only java. Even if you know where the browser exe file is installed you will not be able to use it's handle to control it (no pointers in java, different process, different memory area, etc). Sure, you could write one dll and then use it with jni but the final result would not be multplatform ...

Other possible approach would be to inject some keypress but you would be blind about the browser response (you would have to do some ugly screen capture ).

I don't think it is an easy task so IF I were you I would look in the web for some already made dll or library to control the browser.

I know that selenium does some kind of browser control ( http://docs.seleniumhq.org/ )

my 5 cents in 5 minutes.