使用jsoup从特定标签之间的网页中抓取数据

Question

Currently I am developing a program that allows me to collect the most recent 5 fanfiction stories added to my Ao3 (Archive of Our Own) fandom. These stories will then be added to an ArrayList I have set up which will hold fanfiction submissions from the past week. At the end of every week I plan on having the ArrayList's contents be dumped into a textfile that will allow me to paste it into a Reddit post for my subreddit. Now, to prevent duplicates, I wanted to compare the newly parsed stories with stories currently held in the ArrayList.

(Additional info: The bot will check the webpage every 30 minutes)

The part that I'm getting caught up on is the actual parsing of the webpage and getting the content from between the HTML tags.

I looked up CSS Selectors, but I'm still left thoroughly confused, as almost every example was from what seems like an easy website to scrape from, such as IMBD.

From basic research, it looks like within the main body where I'm looking, the stories are all inside an ordered list tag.

<o1 class="work index group">
    <li class="work blurb group" id="work_10504812" role="article>...</li>
    <li class="work blurb group" id="work_9656693" role="article>...</li>
    <li class="work blurb group" id="work_11814486" role="article>...</li>
    //Goes on for ~20 more stories
    <li class="work blurb group" id="work_11687247" role="article>...</li>
</ol>

So for clarity's sake, each list type is a single story located within the ordered list. Any within one list tag is the following. (ordered list tag added for context)

<ol class="work index group">
    <li class="work blurb group" id="work_10504812" role="article">
  <!--title, author, fandom-->
  <div class="header module">
    <h4 class="heading">
      <a href="/works/10504812">Pocket Healer</a>
      by

      <!-- do not cache -->
      <a rel="author" href="/users/OverNoot/pseuds/OverNoot">OverNoot</a> 
    </h4>
    <h5 class="fandoms heading">
      <span class="landmark">Fandoms:</span>
      <a class="tag" href="/tags/Overwatch%20(Video%20Game)/works">Overwatch (Video Game)</a>
      &nbsp;
    </h5>
    <!--required tags-->
    <ul class="required-tags">
<li> <a class="help symbol question modal modal-attached" title="Symbols key" aria-controls="#modal" href="/help/symbols-key.html"><span class="rating-general-audience rating" title="General Audiences"><span class="text">General Audiences</span></span></a></li>
<li> <a class="help symbol question modal modal-attached" title="Symbols key" aria-controls="#modal" href="/help/symbols-key.html"><span class="warning-no warnings" title="No Archive Warnings Apply"><span class="text">No Archive Warnings Apply</span></span></a></li>
<li> <a class="help symbol question modal modal-attached" title="Symbols key" aria-controls="#modal" href="/help/symbols-key.html"><span class="category-femslash category" title="F/F"><span class="text">F/F</span></span></a></li>
<li> <a class="help symbol question modal modal-attached" title="Symbols key" aria-controls="#modal" href="/help/symbols-key.html"><span class="complete-no iswip" title="Work in Progress"><span class="text">Work in Progress</span></span></a></li>
</ul>
    <p class="datetime">17 Aug 2017</p>
  </div>
  <!--warnings again, cast, freeform tags-->
  <h6 class="landmark heading">Tags</h6>
  <ul class="tags commas">
    <li class="warnings"><strong><a class="tag" href="/tags/No%20Archive%20Warnings%20Apply/works">No Archive Warnings Apply</a></strong></li><li class="relationships"><a class="tag" href="/tags/Fareeha%20%22Pharah%22%20Amari*s*Angela%20%22Mercy%22%20Ziegler/works">Fareeha "Pharah" Amari/Angela "Mercy" Ziegler</a></li><li class="characters"><a class="tag" href="/tags/Fareeha%20%22Pharah%22%20Amari/works">Fareeha "Pharah" Amari</a></li> <li class="characters"><a class="tag" href="/tags/Angela%20%22Mercy%22%20Ziegler/works">Angela "Mercy" Ziegler</a></li> <li class="characters"><a class="tag" href="/tags/Winston%20(Overwatch)/works">Winston (Overwatch)</a></li> <li class="characters"><a class="tag" href="/tags/Lena%20%22Tracer%22%20Oxton/works">Lena "Tracer" Oxton</a></li><li class="freeforms"><a class="tag" href="/tags/Tiny%20Pharah%20and%20Tiny%20Mercy/works">Tiny Pharah and Tiny Mercy</a></li> <li class="freeforms"><a class="tag" href="/tags/Fluff/works">Fluff</a></li> <li class="freeforms last"><a class="tag" href="/tags/Cute/works">Cute</a></li>
  </ul>
  <!--summary-->
    <h6 class="landmark heading">Summary</h6>
    <blockquote class="userstuff summary">
      <p>Angela and Fareeha wake up to find tiny alternate versions of themselves have appeared and are now imprinted on them. How will these tiny Pharahs and Mercies impact their work at Overwatch and more importantly how will it impact the feelings they have for each other.</p>
    </blockquote>
  <!--stats-->

  <dl class="stats">
      <dt class="language">Language:</dt>
      <dd class="language">English</dd>
    <dt class="words">Words:</dt>
    <dd class="words">35,143</dd>
    <dt class="chapters">Chapters:</dt>
    <dd class="chapters">10/11</dd>
    <dt class="comments">Comments:</dt>
    <dd class="comments"><a href="/works/10504812?show_comments=true&amp;view_full_work=true#comments">168</a></dd>
    <dt class="kudos">Kudos:</dt>
    <dd class="kudos"><a href="/works/10504812?view_full_work=true#comments">438</a></dd>
    <dt class="bookmarks">Bookmarks:</dt>
    <dd class="bookmarks"><a href="/works/10504812/bookmarks">35</a></dd>
    <dt class="hits">Hits:</dt>
    <dd class="hits">5890</dd>
  </dl>
</li>

And basically I wanted to extract the title, author, url, summary, and rating.

So far I've gathered the locations of the items I want to extract, but I have no actual idea how to do so.

Title:

<a href="/works/10504812">Pocket Healer</a>

Author:

<a rel="author" href="/users/OverNoot/pseuds/OverNoot">OverNoot</a>

Url:

<li class="work blurb group" id="work_10504812" role="article">
<!--(http://archiveofourown.com/works/<the number after 'work_'>)-->

Summary:

<blockquote class="userstuff summary">
    <p> (SUMMARY GOES HERE) </p>
</blockquote>

Rating:

<li> <a class="help symbol question modal modal-attached" title="Symbols key" aria-controls="#modal" href="/help/symbols-key.html"><span class="rating-general-audience rating" title="General Audiences"><span class="text">General Audiences</span></span></a></li>

Additional question: Is it possible to iterate through the contents of the ordered list in something like a forloop?

The current code I have set up for opening the webpage is below.

    while (true) {
        try {

            String url = "http://archiveofourown.org/tags/Fareeha%20%22Pharah%22%20Amari*s*Angela%20%22Mercy%22%20Ziegler/works";
            Document doc = Jsoup.connect(url).get();

            //Returns element of webpage
            doc.select("<Narrow down to ordered list>");

            //Run for loop to run through first 5 items of 
            Thread.sleep(THIRTY_MINUTES);

        }
        catch (Exception ex) {
            ex.printStackTrace();
        }

    }

Answer 1

You can use Document.select(String cssSelector) method that returns Elements that you can iterate over. For example ol.work > li will return all li elements that are first-level children to this ol.work element. You can use this to iterate over all stories.

Consider following part of code:

Elements ol = doc.select("ol.work > li");

for (Element li : ol) {
    String title = li.select("h4.heading a").first().text();
    String author = li.select("h4.heading a[rel=author]").text();
    String id = li.attr("id").replaceAll("work_","");
    String url = "http://archiveofourown.com/works/" + id;
    String summary = li.select("blockquote.summary").text();
    String rating = li.select("span.rating").text();

    System.out.println("Title: " + title);
    System.out.println("Author: " + author);
    System.out.println("ID: " + id);
    System.out.println("URL: " + url);
    System.out.println("Summary: " + summary);
    System.out.println("Rating: " + rating);
}

In this example we get all li elements in for-loop and extract expected content. As you can see we use select method for every data extraction limited to current li element. Element.text() method returns body of an element as a plain text, removing all tags if they are present.

Running following code with HTML you put in your question produces following output:

Title: Pocket Healer
Author: OverNoot
ID: 10504812
URL: http://archiveofourown.com/works/10504812
Summary: Angela and Fareeha wake up to find tiny alternate versions of themselves have appeared and are now imprinted on them. How will these tiny Pharahs and Mercies impact their work at Overwatch and more importantly how will it impact the feelings they have for each other.
Rating: General Audiences

I hope it helps.