I am trying to extract comments from few forums from the website. I have list of links from which I want the comments to be extracted. When I give single link instead of {i} in the code (f"{i}/index{item}/")) the code works fine, but with the below code it is giving an empty list.
data
name Link
a https://www.f150forum.com/f118/2019-adding-ada...
b https://www.f150forum.com/f118/2018-adding-ada...
c https://www.f150forum.com/f118/adaptive-cruise...
d https://www.f150forum.com/f118/2018-platinum-s...
e https://www.f150forum.com/f118/adaptive-cruise...
f https://www.f150forum.com/f118/adaptive-cruise...
My code
link_url = []
username=[]
comments = []
for i in df['Link']:
with requests.Session() as req:
for item in range(1):
r = req.get(
f"{i}/index{item}/")
soup = BeautifulSoup(r.text, 'html.parser')
link_url.append(item)
for item in soup.findAll('div',attrs={"class":"ism-true"}):
result = [item.get_text(strip=True, separator=" ")]
comments.append(result)
for item in soup.findAll('a',attrs={"class":"bigusername"}):
name = [item.get_text(strip=True, separator=" ")]
username.append(name)
Can you please help me with this. Thank you in advance.
OK , I see your links are in a dataframe, you can loop them with :
import pandas as pd
from io import StringIO
data = """
name,Link
a,https://www.f150forum.com/f118/2019-adding-ada...
b,https://www.f150forum.com/f118/2018-adding-ada...
c,https://www.f150forum.com/f118/adaptive-cruise...
d,https://www.f150forum.com/f118/2018-platinum-s...
e,https://www.f150forum.com/f118/adaptive-cruise...
"""
df = pd.read_csv(StringIO(data),sep=',')
for index, row in df.iterrows():
print(row['Link'])
result :
https://www.f150forum.com/f118/2019-adding-ada...
https://www.f150forum.com/f118/2018-adding-ada...
https://www.f150forum.com/f118/adaptive-cruise...
https://www.f150forum.com/f118/2018-platinum-s...
https://www.f150forum.com/f118/adaptive-cruise...
then , put the value(link) inside of your requests
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.