Jsoup html解析span下的项目

Question

I'm trying to parse items using Jsoup on Java. When i open the source code for the website i'm trying to read

    <ul class="myptab typ3">
        <li><span class="active"><a href="#;" id="ONE_TO_ONE">1+1</a></span></li>
        <li><span><a href="#;" id="TWO_TO_ONE">2+1</a></span></li>
    </ul>


    <h5 class="invisible" >1+1 itemlist</h5>
    <div class="tblwrap mt50">
        <ul class="prod_list">
        </ul>
        <div class="paging">
        </div>
    </div>

    <h5 class="invisible">2+1 itemlist</h5>
    <div class="tblwrap mt50">
        <ul class="prod_list">

        </ul>
        <div class="paging">
        </div>
    </div>

In this source code the for either 1+1 section or 2+1 section is listed but when i use inspect to see the source code on the 1+1 section item,

    <h5 class="invisible">1+1 itemlist</h5>
    <div class="tblwrap mt50"> ==$0
    <ul class="prod_list">
      <li>
        <div class="prod_box">
          <p class="img"></p>
          <p class="title">mangomilk_pet_300ML</p>
      </li>
      <li>...</li>
      <li>...</li>
      <li>...</li>
    </ul>

it pops up like that. so I'd like to select p.title and p.img from the hidden span items in the source code.

Answer 1

It looks like the content is built dynamically by javascript. In this case jsoup is not enough. You can try to use jBrowserDriver which is able to retrieve completed DOM (with rendered javascript part).

Example code:

// Represents result item
public class ProductBox {
    private final String image;
    private final String title;

    public ProductBox(String image, String title) {
        this.image = image;
        this.title = title;
    }

    public String getImage() { return image; }
    public String getTitle() { return title; }
}


// Method responsible for parsing a page
public void processPage(String url) {
    JBrowserDriver driver = new JBrowserDriver(Settings
            .builder()
            .timezone(Timezone.AMERICA_NEWYORK)
            .userAgent(UserAgent.CHROME)
            .build());

    driver.get(url);
    String pageSource = driver.getPageSource();
    driver.quit();

    Document doc = Jsoup.parse(pageSource);
    Elements prodBoxes = doc.select("ul.prod_list div.prod_box");
    List<ProductBox> products = prodBoxes.stream()
            .map(e -> new ProductBox(e.select("p.img").text(), e.select("p.title").text()))
            .collect(Collectors.toList());

    products.forEach(e -> System.out.printf("%s - %s\n", e.getImage(), e.getTitle()));
}