How to get a JS redirected pdf linked from a web page

Question

I am using requests to get web pages, for example as follows.

import requests
from bs4 import BeautifulSoup
url = "http://www.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/CARE/EY298883"
r = requests.get(url)
soup = BeautifulSoup(r.text)

For each one of these pages I would like to get the first pdf that is point to in the section titled "Latest reports". How can you do this with beautiful soup?

The relevant part of the HTML is

 <tbody>
 <tr>
          <th scope="col">Latest reports</th>
          <th scope="col" class="date">Inspection <br/>date</th>
          <th scope="col" class="date">First<br/>publication<br/>date</th>
          </tr>
          <tr>
            <td><a href="/provider/files/1266031/urn/106428.pdf"><span class="icon pdf">pdf</span> Early years inspection report </a></td>
            <td class="date">12 Mar 2009</td>
            <td class="date">4 Apr 2009</td>
            </tr>        </tbody>

---

The following code looks like it should work but does not.

 ofstedbase = "http://www.ofsted.gov.uk" for col_header in soup.findAll('th'): if not col_header.contents[0] == "Latest reports": continue for link in col_header.parent.parent.findAll('a'): if 'href' in link.attrs and link['href'].endswith('pdf'): break else: print '"Latest reports" PDF not found' break print '"Latest reports" PDF points at', link['href'] p = requests.get(ofstedbase+link['href']) print p.content break 

The problem is that p contains another web page and not the pdf it should. Is there some way to get the actual pdf?

---

Update:

Got it to work with one more iteration of BeautifulSoup

 souppage = BeautifulSoup(p.text)
 line = souppage.findAll('a',text=re.compile("requested"))[0]
 pdf = requests.get(ofstedbase+line['href'])

Any better/nicer solutions gratefully received.

Answer 1

It's not the cleanest solution, but you can iterate through column headers until you find "Latest reports", then search that table for the first link that points at a PDF file.

for col_header in soup.findAll('th'):
    if not col_header.contents[0] == "Latest reports": continue
    for link in col_header.parent.parent.findAll('a'):
        if 'href' in link.attrs and link['href'].endswith('pdf'): break
    else:
        print '"Latest reports" PDF not found'
        break
    print '"Latest reports" PDF points at', link['href']
    break

You might try Selenium WebDriver ( python -m "easy_install" selenium ) to automatically instruct Firefox to download the file. This requires Firefox:

from selenium import webdriver
from bs4 import BeautifulSoup

profile = webdriver.FirefoxProfile()
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', ('application/pdf'))
profile.set_preference("pdfjs.previousHandler.alwaysAskBeforeHandling", False)
profile.set_preference("browser.helperApps.alwaysAsk.force", False)
profile.set_preference("browser.download.manager.showWhenStarting", False)

driver = webdriver.Firefox(firefox_profile = profile)
base_url = "http://www.ofsted.gov.uk"
driver.get(base_url + "/inspection-reports/find-inspection-report/provider/CARE/EY298883")
soup = BeautifulSoup(driver.page_source)

for col_header in soup.findAll('th'):
    if not col_header.contents[0] == "Latest reports": continue
    for link in col_header.parent.parent.findAll('a'):
        if 'href' in link.attrs and link['href'].endswith('pdf'): break
    else:
        print '"Latest reports" PDF not found'
        break
    print '"Latest reports" PDF points at', link['href']
    driver.get(base_url + link['href'])

This solution is very powerful because it can do everything a human user can, but it has drawbacks. For example, I've tried to address the issue of Firefox prompting for the download, but it doesn't work for me. Results may vary depending on your installed add-ons and Firefox version.

Answer 2

Got it to work with one more iteration of BeautifulSoup

 souppage = BeautifulSoup(p.text)
 line = souppage.findAll('a',text=re.compile("requested"))[0]
 pdf = requests.get(ofstedbase+line['href'])