美丽的汤和解析 reddit

Question

just been trying to parse reddit shower thoughts for the submissions and have run into a problem:

 path = 'https://www.reddit.com/r/Showerthoughts/'

 with requests.Session() as s:

    r = s.get(path)
    soup = BeautifulSoup(r.content, "lxml")

    # print(soup.prettify())

    threads = soup.find_all('p')


    for thread in threads:
        soup = thread
        text = soup('a')
        try:
            print(text[0])
        except:
            pass

in this code I am trying to just get the title of each submission which is enclosed in a < p > tag and then an < a > tag with class "title may-blank". but above code returns all elements with an a tag of which there are many and even thought the titles are there I would have to go through two more interations of soup.findAll() , and I am sure there is a less manual way of searching through the soup to print all of the titles

from my knowledge I tried to do

titles = soup.findAll( "a", {"class":"title may-blank}) for title in titles: print(title.string)
but this didnt work any thoughts? PS I know this can be done with the reddit API and is more efficient, but I want to improve my parsing skills because they are not up to scratch. Thank you for the help

Answer 1

They are css classes, also you need to add a user-agent:

import requests
from bs4 import BeautifulSoup
path = 'https://www.reddit.com/r/Showerthoughts/'
headers ={"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"}
with requests.Session() as s:
    r = s.get(path, headers=headers)
    soup = BeautifulSoup(r.content, "lxml")
    threads = soup.select('a.title.may-blank')
    for a in threads:
        print(a)

You could also use soup.find_all("a", class_="title") but that could match more than you want.