美汤——选择Class有意外结果

Question

I am new to programming and have been learning Python through web scraping. What I am trying to do is capture the below line from the site listed in my URL:

<a class="" href="https://www.adweek.com?paged=776%3Fs%3Dinterpublic&amp;orderby=date&amp;s=interpublic">776</a> , but I cannot seem to get there. It only returns the first line of pagination information and I can't figure out why. Any help would be greatly appreciated

import requests
from bs4 import BeautifulSoup
url = 'https://www.adweek.com/?s=interpublic&orderby=date'
req = requests.get(url)
soup = BeautifulSoup(req.content, 'html.parser')
k =soup.find_all('div', {'class':'pagination-centered'})

Returns only --

[<div class="pagination-centered"><ul class="pagination">
 <li><span aria-current="page" class="current">1</span></li></ul></div>]

Thanks, Seth

Answer 1

You can get pagination using a[href*="paged="] css selector:

import requests
from bs4 import BeautifulSoup

url = 'https://www.adweek.com/?s=interpublic&orderby=date'
req = requests.get(url)
soup = BeautifulSoup(req.content, 'html.parser')

# print text and href
pagination = soup.select('a[href*="paged="]')
for p in pagination:
    print(p.text.strip(), p.get('href'))

"Next" has same url as first link, you can use set to get only unique href. :

pagination = {p['href'] for p in soup.select('a[href*="paged="]')}

You can get last page number and iterate by changing parameter paged in the url until the last page.

Page source without JavaScript: