使用jsoup抓取Google广告

Question

I used jsoup a few months back to scrape all the google search results , not including ads. now comes the exact opposite for the job , i need to get all of the ads from google results. the thing is i cant find them in my document.

problem is surely a wrong tag...

Elements elements = doc.select("div[class=*What do i need to put here?*]");
                    for (Element link : elements) {
                        position++;

                        Elements tempTitles = link.select("h3[]");
                        Element tempSmtng = link.select("a").first();
                        .............

this is a code taken from that last job. it used to say class=g and worked great but now it seems like the ads class tag just dosent work. any suggestions what is the tag im looking for?

Answer 1

You should be able to figure this out on your own pretty easily. Just use a browser with developer tools like Chrome and use Inspect Element on the ads. You should see what CSS classes are being implemented.

Details about using Chrome Inspect Element here: https://developers.google.com/web/tools/chrome-devtools/iterate/inspect-styles/?hl=en

JSoup uses CSS selectors to find elements. You can read up on how to use them here: http://css.maxdesign.com.au/selectutorial/ .

You'll be much better off understanding how your code works by learning the underlying concepts because webscrapers are inherently brittle since the website provider can change their output whenever they want.