如何使用Jsoup选择在同一树级别存在的两个（或更多）HTML元素？

Question

I'm working on a project and I faced a problem. I need to scrape data from the website that contains following HTML code:

<div class="lin-curso" style="border: 0;">
    <div class="lin-area-c3">
        Vagas 2017
    </div>
</div>
<div class="box10">
    <div class="lin-area-c1">
        L160
    </div>
    <div class="lin-area-c2">
        Acupuntura
    </div>
    <div class="lin-area-c3">
        [Lic-1º cic]
    </div>
</div>
<div class="lin-curso">
    <div class="lin-curso-c1">
        &nbsp;
    </div>
    <div class="lin-curso-c2">
        3155
    </div>
    <div class="lin-curso-c3">
        <a href="detcursopi.asp?codc=L160&amp;code=3155" title="3155/L160">Instituto Politécnico de Setúbal - Escola Superior de Saúde</a>
    </div>
    <div class="lin-curso-c4">
        20
    </div>
</div>
<br>
<div class="box10">
    <div class="lin-area-c1">
        9059
    </div>
    <div class="lin-area-c2">
        Administração e Gestão de Empresas
    </div>
    <div class="lin-area-c3">
        [Lic-1º cic]
    </div>
</div>
<div class="lin-curso">
    <div class="lin-curso-c1">
        &nbsp;
    </div>
    <div class="lin-curso-c2">
        2270
    </div>
    <div class="lin-curso-c3">
        <a href="detcursopi.asp?codc=9059&amp;code=2270" title="2270/9059">Universidade Católica Portuguesa - Faculdade de Ciências Económicas e Empresariais</a>
    </div>
    <div class="lin-curso-c4">
        n.d.
    </div>
</div>
<br>
<div class="box10">
    <div class="lin-area-c1">
        8056
    </div>
    <div class="lin-area-c2">
        Administração e Gestão Pública
    </div>
    <div class="lin-area-c3">
        [Lic-1º cic]
    </div>
</div>
<div class="lin-curso">
    <div class="lin-curso-c1">
        &nbsp;
    </div>
    <div class="lin-curso-c2">
        4275
    </div>
    <div class="lin-curso-c3">
        <a href="detcursopi.asp?codc=8056&amp;code=4275" title="4275/8056">Instituto Superior de Ciências da Administração</a>
    </div>
    <div class="lin-curso-c4">
        20
    </div>
</div>
<br>
<div class="box10">
    <div class="lin-area-c1">
        8194
    </div>
    <div class="lin-area-c2">
        Administração da Guarda Nacional Republicana
    </div>
    <div class="lin-area-c3">
        [Mest Integ]
    </div>
</div>
<div class="lin-curso">
    <div class="lin-curso-c1">
        &nbsp;
    </div>
    <div class="lin-curso-c2">
        7510
    </div>
    <div class="lin-curso-c3">
        <a href="detcursopi.asp?codc=8194&amp;code=7510" title="7510/8194">Academia Militar</a>
    </div>
    <div class="lin-curso-c4">
        n.d.
    </div>
</div>
<br>
<div class="box10">
    <div class="lin-area-c1">
        9672
    </div>
    <div class="lin-area-c2">
        Administração e Marketing
    </div>
    <div class="lin-area-c3">
        [Lic-1º cic]
    </div>
</div>

BOX10 and line-curso should form an element and they don't. Because in some lines there is only one BOX10 for one Lin-curso but there are lines that are like Lin-curso for one Box10 , if Box10 and Lin-curso were an element there wouldn't be a problem , is there a way I can associate those two ?

EDIT: The website link is this : http://www.dges.gov.pt/guias/indcurso.asp?letra=A

And the element is the ".inside"

Answer 1

Solution to this problem is fairly easy when you use sibling selector. In your case div with class box10 plays role of a header in the table and sibling divs with class lin-curso play role of table data rows. I would suggest firstly selecting all divs with class box10 :

Elements boxes = doc.select("div.box10");

Then you can iterate over boxes and do two major things:

1. Extract data you are interested in from this div (it contains 3 child nodes, divs with classes lin-area-c1 , lin-area-c2 and lin-area-c3 )
2. Select sibling nodes with class lin-curso and extract data from them.

Jsoup provides a method called Element.nextElementSibling() that return sibling element to the element you called this method on. So when you call it on element div.box10 you will get sibling element div.lin-curso .

Sibling in this case means a node immediately following the specified node at the same tree level.

Exemplary solution

Below you can find exemplary code that parses given website and prints table to the console output:

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import java.io.IOException;

final class TestMain {

    public static void main(String[] args) throws IOException {
        Document doc = Jsoup.connect("http://www.dges.gov.pt/guias/indcurso.asp?letra=A").get();

        Elements boxes = doc.select("div.box10");

        for (Element box : boxes) {
            String linAreaC1 = box.select(".lin-area-c1").text();
            String linAreaC2 = box.select(".lin-area-c2").text();
            String linAreaC3 = box.select(".lin-area-c3").text();

            System.out.printf("%s: %s %s%n", linAreaC1, linAreaC2, linAreaC3);

            Element linCurso = box.nextElementSibling();

            while (linCurso.hasClass("lin-curso")) {
                String linCursoC2 = linCurso.select(".lin-curso-c2").text();
                String linCursoC3 = linCurso.select(".lin-curso-c3").text();
                String linCursoC4 = linCurso.select(".lin-curso-c4").text();

                System.out.printf("%s\t%s\t%s%n", linCursoC2, linCursoC3, linCursoC4);

                linCurso = linCurso.nextElementSibling();
            }

            System.out.println("==============================");
        }
    }
}

I hope it helps.