打开并保存网页selenium java

Question

I need to get the content of some web pases like " http://www.ncbi.nlm.nih.gov/nuccore/NM_007002 " for my project. The problem is that I need to open the page from a browser and save it to get the full content (if I try to use the libraries URL and BufferReader I get the "frame" of the page but not the text I need). My professor told me to use Seleniume to open and download the pages I need and then read and parse the relevant information.

Unfortunately, I can't find an example from a JAVA code that open and save a web page. Can anyone explane to my how to do this?

I want to SAVE the page to my computer, not copy the source and save it for file. Not all of the information appears in the source! It's hidden.

Answer 1

In Selenium you can do this:

SafariDriver driver = new SafariDriver(); //you can use any drivers like Chrome,FireFox
driver.get("your link");
String pageSource = driver.getPageSource(); //now you have the page source
//you can save the pageSource to the file or do what ever you want. 

Look at the getPageSource docs here .

If you want to get data from the specific tags, like say for example body , then you can do this:

String pageSource=driver.findElement(By.tagName("body")).getText();

Answer 2

Keep in mind that Selenium is meant for web page automation, so for interacting with the pages automatically. If only the source is really what you need, you can use a JSoup a really solid Java Html parser, in two lines of code, you should have your source

     try {
            Document doc = Jsoup.connect("http://www.ncbi.nlm.nih.gov/nuccore/NM_007002").userAgent("Mozilla/5.0").timeout(30000).get();
            System.out.println(doc.toString());
        } catch (IOException e) {
            e.printStackTrace();
        }