如何从网站 Python 中的所有链接中提取评论

Question

I am trying to extract comments from few forums from the website. I have list of links from which I want the comments to be extracted. When I give single link instead of {i} in the code (f"{i}/index{item}/")) the code works fine, but with the below code it is giving an empty list.

data

    name                    Link
    a               https://www.f150forum.com/f118/2019-adding-ada...
    b               https://www.f150forum.com/f118/2018-adding-ada...
    c               https://www.f150forum.com/f118/adaptive-cruise...
    d               https://www.f150forum.com/f118/2018-platinum-s...
    e               https://www.f150forum.com/f118/adaptive-cruise...
    f               https://www.f150forum.com/f118/adaptive-cruise...

My code

link_url = []
username=[]
comments = []

for i in df['Link']:
    with requests.Session() as req:
        for item in range(1):
            r = req.get(
            f"{i}/index{item}/")
            soup = BeautifulSoup(r.text, 'html.parser')
            link_url.append(item)
            for item in soup.findAll('div',attrs={"class":"ism-true"}):
                result = [item.get_text(strip=True, separator=" ")]
                comments.append(result)
            for item in soup.findAll('a',attrs={"class":"bigusername"}):
                name = [item.get_text(strip=True, separator=" ")]
                username.append(name)

Can you please help me with this. Thank you in advance.

Answer 1

OK , I see your links are in a dataframe, you can loop them with :

import pandas as pd
from io import StringIO

data = """
name,Link
a,https://www.f150forum.com/f118/2019-adding-ada...
b,https://www.f150forum.com/f118/2018-adding-ada...
c,https://www.f150forum.com/f118/adaptive-cruise...
d,https://www.f150forum.com/f118/2018-platinum-s...
e,https://www.f150forum.com/f118/adaptive-cruise...
"""
df = pd.read_csv(StringIO(data),sep=',')
for index, row in df.iterrows():
  print(row['Link'])

result :

https://www.f150forum.com/f118/2019-adding-ada...
https://www.f150forum.com/f118/2018-adding-ada...
https://www.f150forum.com/f118/adaptive-cruise...
https://www.f150forum.com/f118/2018-platinum-s...
https://www.f150forum.com/f118/adaptive-cruise...

then , put the value(link) inside of your requests