JSOUP Web scraping page with loading text and circle

Question

I am web scraping a web page. However, there is a loading text and circle that appear before the store list appears. How do I get the HTML of the store listing which appears after the loading animation?

This is my output -

            <div class="content"> 
             <h1 class="title"> Airdrie store listing(s)</h1> 
             <p class="store-list-intro" data-store-message="There are {0}                                  stores in Airdrie" data-nostore-message="No stores  available in
                                &nbsp;Airdrie"> </p> 
            <div class="loader"> 
            <img src="/etc/designs/lclonline/images/global/loader.gif" alt=""> 
              <p> Loading...</p> 
             </div> 
            </div>

Answer 1

You can't do this with Jsoup. It looks like the content is being loaded asynchronously via JavaScript, and Jsoup can only retrieve the content from the url before being loaded by JavaScript.

You would need a different html client such as HtmlUnit that can actually render JavaScript and wait for the content to load.